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i 

^  G  A  O 

J^^^^^^Accountability  *  Integrity  *  Reliability _ 

United  States  General  Accounting  Office 
Washington,  DC  20548 


October  30,  2001 

The  Honorable  Bob  Schaffer 
House  of  Representatives 

Dear  Mr.  Schaffer: 

Although  the  nation’s  rates  of  violent  juvenile  crime  and  youth 
victimization  have  declined  dramatically  over  the  past  5  years,  critical 
problems  affecting  juveniles  and  the  juvenile  justice  system  still  remain, 
such  as  drug  dependency,  the  spread  of  gangs,  and  child  abuse  and 
neglect.  To  address  these  and  other  issues,  the  Office  of  Juvenile  Justice 
and  Delinquency  Prevention  (OJJDP)  has  funded  a  variety  of 
demonstration,  replication,  research  and  evaluation,  and  training  and 
technical  assistance  programs  aimed  at  preventing  and  responding  to 
juvenile  delinquency  and  juvenile  victimization.  Questions  have  been 
raised,  however,  regarding  what  these  numerous  and  diverse  programs 
have  accomplished. 

You  asked  us  to  review  OJJDFs  msyor  programs,  as  well  as  the 
evaluations  it  has  funded.  Specifically,  we  agreed  to  provide  (1) 
information  on  programmatic  reporting  requirements  for  OJJDP  grantees, 
the  reasons  for  these  requirements,  and  examples  of  the  information 
grantees  have  reported;  (2)  information  on  how  many  juveniles  OJJDP 
grantees  reported  directly  serving  in  fiscal  year  2000,  and  whether  OJJDP 
requires  grantees  to  report  the  number  of  juveniles  they  directly  serve,  and 
if  not,  why;  and  (3)  analysis  of  the  methodological  rigor  of  the  impact 
evaluations  OJJDP  has  funded  of  its  own  programs  since  1995,  and 
information  on  the  other  types  of  evaluations  OJJDP  has  funded.  In 
addition,  you  asked  us  to  provide  information  on  how  much  OJJDP 
awarded  to  various  grant  programs  from  fiscal  years  1996  through  2000 
and  the  types  of  organizations  that  received  these  awards,  and  we  have 
provided  this  information  in  appendix  1.  You  asked  us  how  OJJDP 
disseminates  published  interim  results  of  impact  evaluations  as  well  as 
other  published  publications  OJJDP  and  its  grantees  produce,  and  we  have 
provided  this  information  in  appendix  IL  You  also  asked  us  to  review  how 
OJJDP  monitors  its  grantees,  and  we  have  reported  to  you  on  this  issue.^ 


^  Juvenile  Justice:  Better  Documentation  of  Discretionary  Grant  Monitoring  Is  Needed 
(GAO-02-65,  Oct  10,  2001). 
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Results  in  Brief 


Our  review  of  16  of  OJJDP’s  rricijor  programs  showed  that  although 
virtually  all  grantees  are  required  to  report  on  their  progress  twice  a  year, 
the  information  they  reported  varied.  Grantees  receive  standard,  general 
guidance  for  reporting  on  their  projects  and  providing  OJJDP  information 
with  which  to  monitor  grantees’  projects  and  accomplishments.  According 
to  OJJDP  officials,  such  guidance  needs  to  be  somewhat  general,  given  the 
variation  that  can  occur  among  individual  projects  as  grantees  try  to  meet 
local  needs  and  circumstances.  For  example,  one  Drug-Free  Communities 
Support  Program  grantee  reported  that  it  had  helped  students  produce  an 
anti-smoking  commercial,  while  another  grantee  in  the  same  program 
reported  making  presentations  on  drug  abuse  to  young  men  at  the  local 
juvenile  detention  facility.  Grantees  in  8  of  these  16  programs  are  subject 
to  additional  reporting  requirements  tailored  to  each  of  their  programs, 
and  these  specific  requirements  were  established  primarily  to  support 
outside  evaluations  or  for  other  program  or  project  assessment  purposes. 
For  example,  OJJDP  requires  all  Juvenile  Mentoring  Program  grantees  to 
cooperate  fully  with  an  outside  evaluator.  Grantees  in  this  program  are 
required  to  report  prescribed  demographic  information  for  both  juveniles 
and  mentors  who  participate  in  their  programs,  as  well  as  other  specified 
data. 

We  identified  eight  programs  in  which  all  grantees  reported  the  number  of 
juveniles  they  directly  served.  Grantees  in  these  eight  programs  reported 
serving  about  142,000  juveniles  in  fiscal  year  2000.^  For  example.  Court 
Appointed  Special  Advocate  Program  grantees  reported  serving  over 
70,000  juveniles  through  volunteers  who  advocate  for  the  best  interests  of 
abused  and  neglected  children  who  come  before  the  court.  Juvenile 
Mentoring  Program  grantees  reported  serving  about  8,500  juveniles  by 
providing  one-on-one  mentoring  for  at-risk  youths.  OJJDP  does  not  require 
grantees  in  all  its  programs  to  report  on  the  number  of  juveniles  they  serve 
directly  for  several  reasons,  including  that  many  of  its  programs  are  not 
intended  to  serve  juveniles  directly.  Instead,  OJJDP  programs  have  a 
variety  of  other  purposes  that  can  indirectly  benefit  juveniles,  such  as 
helping  state  and  local  governments  improve  their  juvenile  justice  systems, 
providing  training  and  technical  assistance  to  juvenile  justice 
professionals,  and  establishing  a  centralized  research  effort  on  problems 
of  juvenile  delinquency.  In  commenting  on  a  draft  of  this  report,  the 
Assistant  Attorney  General  stated  that  research  projects  do  not  typically 


^  Grantees  in  three  of  the  eight  programs  reported  data  for  time  periods  that  did  not 
correspond  precisely  with  the  fiscal  year. 
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provide  services  directly,  yet  their  results  can  potentially  help  thousands 
of  juveniles. 

Our  in-depth  review  of  10  of  OJJDP’s  impact  evaluations^  undertaken  since 
1995  of  its  own  programs  raises  some  concerns  about  whether  many  of  the 
evaluations  will  produce  definitive  results.  All  of  these  evaluations  are  stiU 
ongoing,  with  half  in  their  formative  stages  and  half  well  into 
implementation.  While  program  evaluation  is  an  inherently  difficult  task, 
in  some  of  these  evaluations,  the  effort  is  particularly  arduous  because  of 
variations  across  sites  in  how  the  programs  are  implemented.  These 
variations  will  make  it  difficult  to  interpret  evaluation  results.  Two  of  the 
evaluations  that  are  in  their  later  stages  and  three  of  those  that  are  in  their 
formative  stages  lack  specific  plans  for  comparison  groups,  which  would 
aid  in  isolating  the  impacts  of  the  program  from  the  effects  of  other  factors 
that  may  have  influenced  change.  Furthermore,  three  of  the  five 
evaluations  that  are  well  into  implementation  have  developed  data 
collection  problems.  We  are  making  a  recommendation  in  relation  to  the 
five  evaluations  that  are  in  their  formative  stages,  intending  to  ensure  that 
potential  problems  related  to  comparison  groups  and  data  collection  are 
mitigated. 

The  Assistant  Attorney  General  provided  us  with  written  coimnents  on  a 
draft  of  this  report  in  a  letter  dated  October  15,  2001.  In  general,  she 
agreed  with  our  findings  and  recommendation.  However,  she  disagreed 
with  our  focus  on  the  use  of  comparison  groups  as  the  only  valid 
evaluation  design.  Although  she  agreed  that  the  inclusion  of  comparison 
groups  would  strengthen  the  interpretation  of  evaluation  results,  she  said 
that  sufficient  funds  are  not  available  to  include  comparison  groups  in 
every  evaluation  and  that  laboratory-like  comparison  groups  may  not  be 
possible  when  evaluating  real  life  community-based  programs.  We 
recognize  that  not  all  evaluation  issues  that  can  compromise  results  are 
resolvable,  including  the  use  of  comparison  groups.  However,  the  validity 
of  the  evaluation  results  can  be  enhanced  through  establishing  and 
tracking  comparison  groups.  If  other  ways  exist  to  effectively  isolate  the 
impacts  of  a  program,  comparison  groups  may  not  be  needed.  However,  in 
the  10  evaluations  for  which  we  assessed  methodological  rigor,  we  saw  no 
evidence  of  other  methods  being  used.  While  studies  that  do  not  have 
appropriate  comparison  groups  can  provide  useful  information,  they 


^  Impact  evaluations  assess  the  extent  to  which  a  program  caxjses  changes  in  the  desired 
direction  in  the  target  population. 
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should  not  be  considered  impact  evaluations.  The  full  text  of  the  Assistant 
Attorney  Gleneral’s  comments  and  our  evaluation  of  them  are  presented  in 
appendix  IX  and  elsewhere  in  this  report,  as  appropriate. 


Background 


OJJDP,  one  of  the  components  of  the  U.S.  Department  of  Justice,  Office  of 
Justice  Programs  (OJP),  was  established  by  the  Juvenile  Justice  and 
Delinquency  Prevention  Act  of  1974  (Juvenile  Justice  Act)/  Its  mission  is 
to  provide  national  leadership,  coordination,  and  resources  to  prevent  and 
respond  appropriately  to  juvenile  delinquency  and  juvenile  victimization. 
OJJDP  accomplishes  its  mission  through  developing  and  implementing 
prevention  programs  and  a  juvenile  justice  system  that  protects  the  public 
safety,  holds  juvenile  offenders  accountable,  and  provides  treatment  and 
rehabilitative  services  based  on  the  needs  of  juveniles  and  their  famihes. 
OJJDP  funds  research  and  evaluation  efforts,  statistical  studies,  and 
demonstration  programs;  provides  technical  assistance  and  training; 
produces  and  distributes  publications  and  other  products  containing 
information  about  juvenile  justice  topics;  oversees  activities  dealing  with 
missing  and  exploited  children;  and  administers  a  wide  variety  of  grant 
programs.  OJJDP  funds  programs  that  serve  juveniles  directly  as  weU  as 
those  that  benefit  juveniles  more  indirectly  by  focusing  on  system-wide 
changes  or  by  increasing  the  capacity  of  governmental  units  or 
organizations. 

OJJDP  awards  grants  to  states,  territories,  localities,  and  private 
organizations  through  five  formula  and  block  grant  (formula/block  grant) 
programs  and  numerous  discretionary  grant  programs.  OJJDP  administers 
four  formula  grant  programs®  that  provide  funds  directly  to  states  and 
territories  on  the  basis  of  states’ juvenile  populations,  and  one  block  grant 
program®  that  awards  a  fixed  level  of  funds  to  all  states  and  territories. 
Under  these  formula/block  grant  programs,  states  may,  in  turn,  make 
subawards  to  other  organizations  such  as  units  of  local  government. 
OJJDP  awards  discretionary  grants  through  a  competitive  process  to  state 


U2  U.S.C.  5601,  etseq. 

®  OJJDP’s  four  formula  grant  programs  are  the  Formula  Grants  Program,  the  Community 
Prevention  Grants  Program,  State  Challenge  Activities,  and  the  Juvenile  Accountability 
Incentive  Block  Grants  Program. 

®  OJJDP’s  one  block  grant  program  is  the  Enforcing  the  Underage  Drinking  Laws  Program. 
The  Enforcing  the  Underage  Drinking  Laws  Program  has  both  a  block  grant  component 
and  a  discretionary  grant  component. 
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OJJDP’s  Fiscal  Year  2000 
Awards 


governments,  local  governments,  or  individual  agencies  and  organizations. 
OJP  is  responsible  for  the  financial  monitoring^  of  grantees  (i.e.,  it 
provides  policy  guidance,  control,  and  support  services  in  the  financial 
management  of  grants),  whereas  OJJDP  is  responsible  for  program 
monitoring.  For  program  monitoring  purposes,  OJJDP  assigns  each  of  its 
grantees  a  program  manager  who  is  responsible  for  ensuring 
administrative  and  programmatic  compliance  with  relevant  statutes, 
regulations,  policies  and  guidelines  of  awarded  grants.  The  program 
manager  is  also  responsible  for  monitoring  grantees’  performance  and 
progress  as  related  to  grantees’  stated  goals  and  objectives. 

OJJDP’s  budget  has  increased  significantly  over  the  last  6  years — ^from 
about  $188  million  in  fiscal  year  1997  to  about  $596  milhon  in  fiscal  year 
2001.  During  this  time,  the  Congress  has  created  new  programs  and 
increased  impropriations  for  some  existing  ones.  The  Congress  has  also 
provided  direction  each  year  regarding  certain  program  areas  OJJDP 
should  fund.  Overall,  in  fiscal  year  2001, 31  percent  of  OJJDP’s  available 
funds*  were  congressionally  earmarked — ^that  is,  set  aside  for  an 
identifiable  grantee,  specified  amount,  and/or  specific  authorized  purpose. 
For  fiscal  year  2001,  $180  million  of  OJJDP’s  funds  were  available  for 
discretionaiy  grant  awards,  of  which  77  percent  was  earmarked. 


OJJDP  awards  the  majority  of  its  funds  to  grantees  in  its  five 
formula/block  grant  programs.  In  fiscal  year  2000,  the  latest  year  for  which 
awards  data  were  available,  OJJDP  awarded  (1)  about  $354  milhon,  or  64 
percent  of  the  total  funds  awarded,  to  these  formula/block  grant  programs 
and  (2)  just  over  $200  million,  or  36  percent  of  the  total  funds  awarded,  to 
a  wide  range  of  discretionary  grant  programs.  The  programs  awarded  the 
most  funds  in  fiscal  year  2000  were  the  Juvenile  Accoimtability  Incentive 
Block  Grants  Program  ($221  million),  the  Formula  Grants  Program  ($70 
million),  the  Community  Prevention  Grants  Program  ($36  million),  the 
Child  Abuse  and  Neglect  Program*  ($32  million),  the  Missing  and 


^  OJP  conducts  financial  monitoring  of  grantees  through  the  Monitoring  Division  of  its 
Office  of  the  Comptroller. 

^  Of  OJJDP’s  $597  miUion  dollar  appropriation,  only  $583  million  were  “available;”  about 
$12.7  million  were  designated  to  support  OJJHDP  administrative  costs  and  $1.3  milhon 
(.0022)  was  rescinded. 

^  This  program  encompasses  several  other  programs  within  OJJDP’s  Child  Protection 
Division  including  the  Court  Appointed  Special  Advocate  Program,  Children’s  Advocacy 
Centers,  the  Model  Courts  Program,  and  the  Safe  Start  Initiative. 
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Exploited  Children  Program'*'  ($32  miUion),  and  the  Drug-Free 
Communities  Support  Program  ($30  million). 

OJJDP  awards  funds  to  a  wide  range  of  recipients,  with  the  majority  of 
awarded  funds  going  to  state  governments.  As  shoAvn  in  figure  1, 67 
percent  of  the  funds  OJJDP  awarded  in  fiscal  year  2000  went  to  states,  20 
percent  to  nonprofit  organizations,  6  percent  to  school  districts  or 
educational  institutions,  and  5  percent  to  local  governments.  However, 
because  many  grantees  make  subawards  to  other  entities,  the  awards  to 
these  grantees  do  not  reflect  the  ultimate  recipients  of  the  funds  OJJDP 
awards.  For  example,  under  the  Formula  Grants  Program,  states  pass 
through  a  minimum  of  two-thirds  of  their  awarded  funds  to  public  and 
private  nonprofit  organizations.  (See  app.  I  for  data  from  fiscal  years  1996 
through  2000  on  (1)  OJJDP  funds  awarded  to  formula/block  grant  versus 
discretionary  grant  programs,  (2)  OJJDP  awards  by  program  area,  (3) 
OJJDP  award  recipients,  and  (4)  OJJDP  formula/block  grant  awards  by 
state.) 


This  program  encompasses  several  other  programs  within  OJJDP’s  Child  Protection 
Division  including  the  National  Clearinghouse  and  Resource  Center  for  Missing  and 
Exploited  ChUdren,  the  Missing  and  Exploited  Children  Training  and  Technical  Assistance 
program,  and  the  Internet  Crimes  Against  Children  Task  Force  Program. 
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Figure  1 :  OJJDP  Award  Recipients,  Fiscal  Year  2000 


Note:  Percentages  do  not  total  100  due  to  rounding. 

"Local  governments  Include  county  governments,  cities,  townships,  and  special  district  governments. 
"■Schools  include  independent  school  districts  and  state  and  private  institutions  of  higher  education. 
"Other  includes  for-profit  organizations,  Individuals,  Indian  Tribes,  and  other  miscellaneous  recipients. 
Source:  GAO  analysis  of  OJP  database. 


Scope  and 
Methodology 


To  identify  prograinmatic  reporting  requirements^^  for  OJJDP  grantees,  the 
reasons  for  these  requirements,  and  examples  of  information  grantees 
have  reported,  we  reviewed  all  5  of  OJJDP’s  formula/block  grant  programs 
and  we  selected  11  of  its  major  discretionary  grant  programs  to  review 
based  on  OJJDP  officials’  input  regarding  which  programs  were  “mgjor” 
(e.g.,  number  of  grantees,  program  funding,  and/or  importance  of  the 
program).  To  identify  grantee  reporting  requirements,  and  the  purpose  of 
these  requirements,  we  met  with  the  OJJDP  program  managers  who 
monitor  each  of  these  16  programs  or  with  other  key  officials.  For  those 


We  did  not  review  requirements  associated  with  OJP’s  financial  monitoring  of  OJJDP 
grantees. 
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programs  in  which  OJJDP  funded  an  outside  evaluation,  we  also  met  with 
program  managers  who  oversee  the  evaluations.  We  reviewed  OJP’s  Grant 
Management  Policies  and  Procedures  Manual  (January  19,  2001)  and 
Categorical  Assistance  Progress  Report  (progress  report)  form^^  along  with 
the  instructions  for  completing  the  form.  (See  app.  HI  for  a  copy  of  the 
progress  report  form.)  We  also  reviewed  OJJDP  program  documents  for 
each  of  the  16  programs,  including  any  special  reporting  conditions.  To 
supplement  program  documents,  we  reviewed  relevant  documents  from 
outside  evaluators.  We  did  not  assess  the  adequacy  of  reporting 
requirements  established  by  OJJDP  or  the  outside  evaluators. 

To  identify  specific  examples  of  performance  data  that  grantees  reported, 
we  asked  OJJDP  officials  to  provide  progress  reports  for  each  program 
demonstrating  a  range  of  detail  and,  in  some  cases,  we  asked  for  reports 
from  specific  grantees.  For  each  program,  we  then  reviewed  3  to  15 
progress  reports  (or  individual  performance  reports)  submitted  between 
1998  and  2001.  We  did  not  review  progress  reports  from  all  grantees  in 
every  program,  nor  did  we  review  grantees’  compliance  with  reporting 
requirements.  For  programs  being  evaluated  by  an  outside  evaluator,  we 
reviewed  performance  data  that  program  grantees  reported  to  those 
evaluators,  when  available. 

To  determine  whether  OJJDP  requires  grantees  to  report  the  number  of 
juveniles  they  serve  directly  and  to  identify  the  number  of  juveniles  OJJDP 
grantees  reported  serving  in  fiscal  year  2000,  we  interviewed  OJJDP 
program  managers  for  each  of  the  16  programs  we  reviewed  and  examined 
relevant  program  documents,  including  selected  semiannual  progress 
reports.  To  determine  whether  other  programs  directly  serve  juveniles, 
obtain  available  data  on  the  numbers  of  juveniles  served,  and  learn  why 
OJJDP  does  or  does  not  require  grantees  to  report  such  data,  we  reviewed 
OJJDP  program  literature  and  met  with  OJJDP  division  directors. 
Nevertheless,  the  list  of  OJJDP  programs  we  identified  as  directly  serving 
juveniles  may  not  be  comprehensive.  We  focused  our  data  collection  effort 
on  only  those  programs  we  identified  in  which  all  grantees  reported 
juveniles-served  data. 


For  the  Formula  Grants  Program,  we  reviewed  OJJDP’s  instructions  for  completing  the 
annual  Performance  Report,  including  the  Individual  Project  Report  forms,  because 
Formula  Grants  grantees  are  required  to  complete  these  reports  in  lieu  of  progress  reports. 
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To  assess  the  methodological  rigor  of  the  impact  evaluations  OJJDP  has 
funded  since  1995  of  its  own  programs,  and  to  provide  information  on  the 
other  types  of  evaluations  OJJDP  has  funded,  we  asked  OJJDP  to  identify 
all  program  evaluations  it  had  funded  since  1995.  For  each  evaluation,  we 
asked  OJJDP  to  indicate  whether  it  was  an  impact  evaluation,  whether  the 
program  being  evaluated  was  funded  by  OJJDP,  and  whether  the 
evaluation  had  been  completed  or  was  ongoing.  Overall,  OJJDP  identified 
35  evaluations  funded  since  1995. 

Eleven  of  the  35  evaluations  were  impact  evaluations  of  OJJDP  programs, 
and  all  were  ongoing. For  each  of  the  10  impact  evaluations  we  assessed, 
we  asked  OJJDP  to  provide  any  documentation  relevant  to  the  design  and 
implementation  of  the  evaluation  methodologies,  such  as  the  initial  and 
supplemental  proposals,  peer  review  documents,  progress  reports,  reports 
of  interim  results,  and  correspondence  between  OJJDP  and  the  evaluators. 
In  addition,  we  contacted  OJJDP  officials  to  resolve  any  questions  that  we 
had  regarding  the  documentation  and  to  request  any  missing  documents. 
We  did  not  contact  the  program  manager  responsible  for  each  evaluation. 

To  assess  the  methodological  rigor  of  the  10  impact  evaluations,  we  used  a 
data  collection  instrument  to  collect  information  systematically  on  each 
program  being  evaluated  and  the  features  of  the  evaluation  methodology. 
We  based  our  data  collection  and  assessments  on  generally  accepted 
social  science  standards.  We  examined  such  factors  as  whether  evaluation 
data  were  collected  before  and  after  program  implementation,  how 
program  effects  were  isolated  (i.e.,  the  use  of  nonprogram  participant 
comparison  groups  or  statistical  controls),  and  the  appropriateness  of 
sampling,  outcome  measures,  statistical  analyses,  and  any  reported 
results.  A  social  scientist  with  training  and  experience  in  evaluation 
research  and  methodology  read  and  coded  the  documentation  for  each 
evaluation.  A  second  social  scientist  reviewed  each  completed  data 
collection  instrument  and  the  relevant  documentation  for  the  impact 


During  the  course  of  our  review,  OJJDP  officials  identified  10  of  their  35  ongoing 
evaluations  as  impact  evaluations  of  OJJDP-funded  programs.  In  commenting  on  a  draft  of 
this  report,  the  Assistance  Attorney  General  said  that  the  Teen  Courts  evaluation  has  a 
comparison  group  and  should  have  also  been  classified  as  an  impact  evaluation  of  an 
OJJDP-funded  program,  along  with  the  other  10  evaluations.  Because  this  had  not  been 
included  as  an  evaluation  of  an  OJJDP-funded  program  when  we  initially  requested  the 
information,  we  did  not  assess  the  methodological  rigor  of  the  Teen  Courts  evaluation  as 
we  did  with  the  other  10  impact  evaluations.  On  the  basis  of  comments  provided  by  the 
Assistance  Attorney  General,  we  have  reclassified  the  Teen  Courts  evaluation  as  an  impact 
evaluation  of  an  OJJDP-funded  evaluation. 
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evaluation  to  verify  the  accuracy  of  every  coded  item.  We  relied  on 
documents  OJJDP  provided  to  us  in  April  2001  in  assessing  the  evaluation 
methodologies  and  reporting  on  each  evaluation’s  status. 

For  each  of  the  remaining  24  evaluations,  which  included  nonimpact 
evaluations  of  OJJDP-funded  programs,  as  well  as  evaluations  of  juvenile 
justice  programs  that  OJJDP  did  not  fund,  we  asked  OJJDP  to  provide 
general  descriptive  information,  such  as  the  type  and  purpose  of  the 
evaluation,  the  number  of  sites  involved,  and  whether  the  evaluation 
included  data  on  aU  participants.  We  did  not  assess  the  methodological 
rigor  of  these  evaluations. 

We  conducted  our  work  at  OJJDP  Headquarters  in  Washington,  D.C.,  from 
September  2000  to  August  2001  in  accordance  with  generally  accepted 
government  auditing  standards. 


All  Grantees  Receive 
Standard  Reporting 
Guidance  and  Some 
Are  Required  to 
Report  Specific  Data 


OJP  requires  virtually  all  OJJDP  grantees^^  to  submit  semiannual  progress 
reports,  which  OJJDP  uses  to  help  monitor  grantees’  project 
implementation  and  achievement  of  the  goals  they  identified  in  their  grant 
applications.  To  this  end,  OJP  provides  grantees  standard,  general 
guidance  on  the  types  of  program  information  they  are  to  report,  such  as 
narrative  information  on  the  status  of  each  of  their  project  goals  and  the 
quantitative  results  of  their  projects.  In  addition  to  this  standard 
requirement,  grantees  for  some  of  OJJDFs  programs  are  subject  to 
additional  reporting  requirements  that  apply  only  to  their  respective 
programs.  Our  review  of  16  major  programs^^  showed  that  grantees  in  8  of 
the  programs  were  required  to  comply  only  with  the  standard  requirement 
for  information,  and  grantees  in  the  other  8  programs  were  required  to 
report  additional  specified  data.  The  specific  reporting  requirements  were 
established  primarily  to  help  evaluate  the  results  of  these  programs.  Table 
1  identifies  the  16  programs  we  reviewed  and  the  reason  for  the  standard 
or  specific  reporting  requirements  for  each  program. 


^^All  discretionary  and  all  formula^lock  grant  program  grantees,  with  the  exception  of 
Formula  Grants  Program  grantees,  must  submit  these  reports. 

We  reviewed  all  5  of  OJJDFs  formula^lock  grant  programs  and  1 1  of  its  discretionaiy 
grant  programs. 
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Table  1:  Reasons  for  Reporting  Requirements  for  16  Selected  Major  OJJDP  Programs 

Reason  for  reporting  requirement 

Program 

Standard 

requirement 

Specific  requirement 

State  Challenge  Activities 

Monitoring 

None 

Community  Prevention  Grants  Program  (Title  V) 

Monitoring 

None 

Juvenile  Accountability  Incentive  Block  Grants  Program 

Monitoring 

None 

Enforcing  the  Underage  Drinking  Laws  Program  (block  grant  component)® 

Monitoring 

None 

Enforcing  the  Underage  Drinking  Laws  Program  (discretionary  grant 
component)® 

Monitoring 

None 

Tribal  Youth  Program 

Monitoring 

None 

Model  Courts  Program 

Monitoring 

None 

National  Clearinghouse  and  Resource  Center  for  Missing  and  Exploited 
Children 

Monitoring 

None 

Formula  Grants  Program^ 

Monitoring 

Statutory  compliance, 

OJJDP  assessment 

Comprehensive  Gang  Initiative 

Monitoring 

Outside  evaluation, 
qrantee  self-assessment 

Rural  Gang  Initiative 

Monitoring 

Outside  evaluation, 
qrantee  self-assessment 

Juvenile  Mentoring  Program  (JUMP) 

Monitoring 

Outside  evaluation 

SafeFutures 

Monitoring 

Outside  evaluation 

Drug-Free  Communities  Support  Program 

Monitoring 

Outside  evaluation 

Internet  Crimes  Against  Children  Task  Force  Program 

Monitoring 

OJJDP  assessment 

Children’s  Advocacy  Centers 

Monitoring 

OJJDP  assessment, 
qrantee  self-assessment 

Note:  Five  programs  in  this  table  are  formula  or  block  grant  programs:  State  Challenge  Activities,  the 
Community  Prevention  Grants  Program,  the  Juvenile  Accountability  Incentive  Block  Grants  Program, 
Enforcing  the  Underage  Drinking  Laws  Program  (block  grant  component),  and  the  Formula  Grants 
Program.  The  remaining  1 1  programs  are  discretionary  grant  programs. 

The  Enforcing  the  Underage  Drinking  Laws  Program  has  both  a  block  grant  program  comporient  and 
a  discretionary  grant  program  component,  and  OJJDP  administers  each  from  separate  organizational 
units. 

‘’Rather  than  reporting  semiannually  through  progress  reports,  grantees  under  this  prograrn  report  on 
their  subgrantees’  projects  annually  through  Individual  Project  Reports.  The  standard  requirements 
for  reporting  are  similar  for  both  types  of  reports. 

Source:  GAO  analysis. 

Our  review  of  selected  progress  reports  from  the  16  programs  showed 
that,  in  all  but  the  Formula  Grants  Program,  grantees  reported  information 
on  the  status  of  their  activities  and  accomplishments  in  response  to  the 
standard  requirements,  although  the  details  they  reported  varied  as  did  the 
projects  themselves.  Grantees  in  the  eight  programs  with  specific 
reporting  requirements  reported  a  variety  of  descriptive  information  and 
performance  data  to  OJJDP  and/or  outside  evaluators.  (See  app.  IV  for  a 


Page  11 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


program  description,  summary  of  reporting  requirements,  and  examples  of 
what  grantees  reported  in  each  of  the  16  programs  we  reviewed.)^® 


Grantees  Receiving 
Standard  Guidance 
Reported  a  Variety  of 
Information  Used  for 
OJJDP  Monitoring 


All  OJJDP  grantees  are  required  to  report  on  their  project  activities  and 
accomplishments  to  OJP  twice  a  year  using  the  Office  of  Justice  Programs’ 
Categorical  Assistance  Progress  Report  (progress  report)  form/^  The  form 
is  unstructured  and  is  to  be  completed  in  narrative  and/or  chart  form.^^  The 
standard  instructions  to  grantees  for  completing  the  form  state  that 
grantees  should  report  information  on  the  status  of  each  of  their  projects’ 
goals  scheduled  to  be  achieved  during  the  reporting  period  and  set  forth  in 
their  grant  application,  including  quantitative  project  results  based  on 
performance  measures.  Grantees  are  also  instructed  to  report  on  actions 
planned  to  resolve  any  implementation  problems  and  request  any 
technical  assistance  they  might  need.  OJJDP  program  managers  are  to  use 
reported  information  to  help  monitor  grantees’  project  implementation. 
OJJDP  officials  explained  that  because  the  progress  report  is  intended  as 
just  one  of  their  monitoring  tools,  this  standard,  general  guidance  meets 
their  basic  oversight  needs.  They  further  explained  that  guidance  needs  to 
be  somewhat  general  given  the  variation  that  can  occur  among  projects  as 
grantees  tailor  them  to  meet  local  needs  and  circumstances.  OJJDP 
encourages  grantees  to  design  projects  that  meet  the  unique  needs  of  their 
own  communities,  and  therefore  grantees  do  not  always  report  on  the 
same  performance  measures. 

Although  OJJDP  program  managers  have  additional  ways  of  keeping 
abreast  of  grantees’  projects,  such  as  phone  calls  and  on-site  visits,  OJJDP 
officials  indicated  they  would  prefer  to  require  and  obtain  more  specific. 


In  addition,  we  aggregated  performance  data  from  grantees  in  OJJDP^s  training  and 
technical  assistance  programs  and  research  programs  because  such  data  are  quite 
significant  in  terms  of  OJJDP’s  outputs,  and  would  not  be  fuUy  captured  if  we  reported  only 
on  the  16  programs  we  selected  for  review.  Although  training  and  technical  assistance 
grantees  and  research  grantees  were  typically  subject  only  to  standard  reporting 
requirements,  they  reported  similar  types  of  quantitative  performance  data.  Thus,  we  were 
able  to  report  comprehensive  output  data,  rather  than  examples  of  output  data,  as  we  did 
for  selected  grantees  in  the  16  programs.  See  appendix  V  for  a  summary  of  performance 
data  from  training  and  technical  assistance  grantees  and  research  grantees. 

As  previously  noted,  the  Formula  Grants  Program  is  the  only  exception.  In  lieu  of  the 
semiannual  Categorical  Assistance  Progress  Report,  Formula  Grants  Program  grantees  are 
required  to  report  annually  on  their  subgrantees’  activities  and  accomplishments  using 
Individual  Project  Reports. 

A  copy  of  the  progress  report  form  can  be  found  in  appendix  m. 
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and  even  more  frequent,  information  through  the  progress  reports  or  other 
reporting  mechanisms.  However,  according  to  these  officials,  they  are 
reluctant  to  impose  additional  reporting  requirements  on  grantees  because 
of  the  Paperwork  Reduction  Act  of  1995,  which  seeks  to  ensure  that 
federal  agencies  balance  their  need  to  collect  information  with  the 
reporting  and  paperwork  burden  they  impose.  Under  the  Act,  federal 
agencies  have  an  obligation  to  keep  the  paperwork  burden  they  impose  as 
low  as  possible,  and  agencies  must  receive  prior  approval  from  the  Office 
of  Management  and  Budget  for  information  collection  requests. 

We  reviewed  selected  progress  reports  that  grantees  from  each  of  the  16 
programs  submitted  to  OJJDP  and  found  that,  in  all  but  the  Formula 
Grants  Program,  grantees  reported  on  the  status  of  their  projects. 

Grantees  reported  input,  output,  or  outcome  data  related  to  the  process, 
implementation,  and/or  accomplishments  of  their  projects.  They  included 
information  such  as  subgrant  awards,  specific  meetings  held,  staff  hired, 
implementation  difficulties,  number  of  project  participants,  and  behavioral 
change  in  youths.  However,  the  particular  information  grantees  reported 
varied,  as  did  their  projects.  This  variation,  coupled  with  the  unstructured 
format  of  the  progress  report,  makes  it  difficult  to  aggregate  reported  data 

Fourteen  of  the  programs  we  reviewed  had  multiple  grantees  and  the 
information  these  grantees  reported  in  response  to  the  standard  guidance 
varied,  even  within  each  program.  For  example,  we  foimd  the  following: 

•  Under  the  Tribal  Youth  Program— a  program  that  recognizes  differences 
among  tribes  and  encourages  diversity  in  their  projects — grantees  must 
implement  projects  in  keeping  with  at  least  one  of  four  broad  purpose 
areas.  One  tribe  reported  that  it  had  completed  the  renovation  of  a  youth 
center;  another  reported  that  it  had  collected  examples  of  other  tribes’ 
juvenile  law  enforcement  codes^®  and  started  drafting  model  codes 


Funds  are  available  for  projects  that  support  the  following  four  broad  purposes:  (1)  to 
reduce,  control,  and  prevent  crime  by  and  against  tribal  youth;  (2)  to  provide  interventions 
for  court-involved  tribal  youth;  (3)  to  improve  tribal  juvenile  justice  systems;  and  (4)  to 
provide  prevention  programs  focusing  on  alcohol  and  drugs. 

To  address  the  severe  juvenile  crime  problem  in  Indian  commumties,  many  tribal 
governments  have  developed  juvenile  codes  that  typically  include  provisions  relating  to  the 
disposition  of  cases  involving  Indian  youth  arrested  or  detained  by  the  Bureau  of  Indian 
Affairs  or  tribal  law  enforcement  personnel.  Tribal  juvenile  codes  are  generally  tailored  to 
Indian  youth  by  incorporating  traditionally  accepted  cultural  methods  of  dealing  with 
juvenile  delinquency  issues. 
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adapted  for  each  of  its  villages.  A  third  tribe  reported  that  the  resignation 
of  its  community  truancy  officer  had  impacted  its  ability  to  reduce 
instances  of  misbehavior  in  school. 

•  Under  another  program — ^the  Juvenile  Accountability  Incentive  Block 
Grants  Program — ^grantees  (states)  and  their  subgrantees  (communities) 
have  undertaken  a  variety  of  projects  and,  thus,  report  different 
information.  In  this  program,  states  and  their  communities  can  choose 
from  12  different  purpose  areas^^  under  the  broad  objective  of  promoting 
greater  accountability  in  the  juvenile  justice  system.  Thus,  one  state 
reported  that  one  community  hired  a  juvenile  court  intake  officer  and 
included  that  officer’s  caseload;  the  same  state  reported  that  another 
community  was  unable  to  start  a  project  because  a  local  agency  declined 
to  participate  in  its  project.  Another  state  reported  on  the  number  of 
youths  enrolled  in  one  community’s  drug  testing  project  and  reported  the 
number  of  drug  screening  tests  performed. 

•  In  the  Drug-Free  Communities  Support  Program,  grantees  design  projects 
to  meet  the  needs  of  their  local  communities;  thus,  the  projects  and  the 
information  grantees  reported  varied.  For  example,  one  grantee  reported 
that  it  helped  local  students  produce  a  30-second  anti-smoking  commercial 
in  collaboration  with  the  local  health  department  and  further  reported  that 
only  9  of  50  invited  members  attended  a  strategic  planning  meeting  it  had 
held.  Another  grantee  reported  making  presentations  on  drug  abuse  to  146 
young  men  at  the  local  juvenile  detention  facility,  and  that  its  pre-  and 
post-assessments  continually  showed  that  the  young  men  gained 
knowledge  in  the  harmful  effects  of  alcohol  and  drugs. 

In  the  Formula  Grants  Program,  not  all  grantees  reported  on  the  objectives 
and  accomplishments  of  their  subgrantees’  projects,  as  required.  OJJDP 
requires  grantees  in  this  program  to  complete  an  Individual  Project  Report 


Funding  is  available  to  support  the  following  12  purpose  areas:  (1)  construction  of 
juvenile  detention  or  correctional  facilities,  including  training  of  personnel;  (2) 
accountability-based  sanctions  programs;  (3)  hiring  of  judges,  probation  officers,  and 
defenders,  and  funding  pretrial  services;  (4)  hiring  of  prosecutors;  (5)  funding  of 
prosecutor-led  drug,  gang,  and  violence  programs;  (6)  providing  technology,  equipment, 
and  training  programs  for  prosecutors;  (7)  probation  programs;  (8)  gun  courts;  (9)  drug 
courts;  (10)  interagency  information-sharing  systems;  (11)  accoimtability-based  programs 
for  law  enforcement  referrals  or  those  designed  to  protect  students  and  school  personnel 
from  youth  violence;  and  (12)  controlled  substance  testing  (including  interventions)  for 
juveniles  in  the  juvenile  justice  system. 
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(IPR)^^  for  each  subgrantee.  The  instructions  for  completing  IPRs  are 
similar  to  the  instructions  grantees  in  other  programs  receive  for 
completing  semiannual  progress  reports.  Our  review  of  IPRs  from  selected 
states  showed  that,  for  one  state,  none  of  the  IPRs  contained  any 
information  on  subgrantees’  accomplishments,  and  some  did  not  include 
information  on  subgrantees’  program  objectives.^^  For  another  state, 
neither  OJJDP  nor  the  state  was  able  to  provide  us  with  copies  of 
completed  IPRs  because  OJJDP’s  automated  reporting  system  for  states 
was  inoperable. 

Two  of  the  16  programs  we  reviewed  had  only  one  grantee  each.  Although 
both  received  only  standard  reporting  guidance,  they  reported  more 
detailed,  quantitative  output  and  outcome  data  than  grantees  in  the  other 
programs  that  received  only  standard  guidance.  In  the  first  program — the 
National  Clearinghouse  and  Resource  Center  for  Missing  and  Exploited 
Children — ^the  grantee  has  voluntarily  reported  detailed  information  in  a 
structxired  format.  In  the  second  program — ^the  Model  Courts  Program — 
OJJDP  has  emphasized  that  the  grantee  should  include  quantitative 
performance  data  in  its  progress  reports,  but  did  not  prescribe  the  specific 
indicators  on  which  the  grantee  must  report, 

OJJDJP  has  designated  the  National  Center  for  Missing  and  Exploited 
Children  (NCMEC)  as  the  grantee  for  National  Clearinghouse  and 
Resource  Center  for  Missing  and  Exploited  Children,  and  NCMEC  has 
developed  its  own  standardized  reporting  format  that  covers  10  categories. 
This  format  collects  numbers  and  other  information  on  each  category, 
such  as  missing  children  cases,  exploited  children  cases,  public  affairs, 
and  hotline  calls.  NCMEC  reports  to  OJJDP  quarterly,  rather  than 
semiannually,  because  this  timeframe  matches  the  reporting  structure  of 
its  data  management  system.  For  the  first  quarter  of  2001,  NCMEC 
reported  various  output  and  outcome  data  that  included  receiving  24,983 
calls  through  its  hotline;  assisting  in  the  recovery  of  1,610  missing 


IPRs  are  part  of  annual  Performance  Reports,  which  Formula  Grants  Program  grantees 
are  required  to  complete. 

Although  assessing  the  quality  of  OJJDP’s  monitoring  efforts  was  outside  the  scope  of 
our  review,  we  inquired  about  the  lack  of  information  in  this  particular  state’s  IPRs. 
According  to  the  responsible  OJJDP  official,  OJJDP  staff  addressed  this  issue  during  a  site 
visit  made  subsequent  to  the  state’s  annual  submission,  and  also  in  a  foUow-up  letter  to  the 
state. 
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children^ ^  receiving  5,291  tips  on  its  online  child  pornography  tip  line;  and 
displaying  pictures  of  1,399  missing  children,  which  resulted  in  locating 
257  children. 

The  sole  grantee  of  the  Model  Courts  Program — ^the  National  Council  of 
Juvenile  and  Family  Court  Judges^^ — also  reports  quantitative  information 
in  its  semiannual  progress  reports.  Although  OJJDP  has  not  specified  the 
performance  indicators  on  which  the  Council  must  report,  it  has 
emphasized  to  the  Council  the  need  for  quantitative  performance  data  in 
the  semiannual  progress  reports.  As  a  result,  the  Council  includes  specific 
quantitative  output  data  in  its  progress  reports.  For  example,  it  reported 
that  during  the  last  half  of  2000  it  distributed  17,818  technical  assistance 
bulletins,  conducted  96  training  presentations,  and  made  31  site  visits  to 
model  courts.  In  addition,  the  Council  voluntarily  publishes  an  annual 
report  that  provides  more  detailed  information  on  the  accomplishments  of 
the  individual  model  courts,  such  as  a  reduction  in  the  number  of  children 
in  court  custody.  OJJDP  officials  told  us  that  they  do  not  require  the 
Council  to  provide  this  report,  but  they  have  instructed  it  to  report 
detailed  performance  data  on  the  activities  of  the  model  courts,  when  such 
data  exist.  They  further  explained  that  if  the  Council  were  to  stop 
publishing  an  annual  report,  OJJDP  would  require  it  to  include  model 
court  performance  data  in  its  progress  reports. 


Grantees  in  Some 
Programs  Are  Required  to 
Report  Specific 
Performance  Data,  Often 
for  Evaluative  Purposes 


In  eight  of  the  major  programs  we  reviewed,  grantees  were  given 
additional,  more  explicit  reporting  instructions  requiring  them  to  report  on 
the  same  specific  performance  measures  as  other  grantees  in  the  same 
program.  In  these  programs,  additional  requirements  were  established  to 
meet  the  evaluative  needs  of  OJJDP  or  an  outside  evaluator.  In  one  of  the 
eight  programs,  requirements  were  also  established  to  ensure  grantee 
compliance  with  certain  requirements  of  the  Juvenile  Justice  Act  as  well 
as  for  program  assessment.  The  specific  requirements  of  each  of  these 


“Missing  children”  includes  nonfamily  and  family  abductions;  runaways  and 
“thrownaways”;  and  lost,  injured,  or  otherwise  missing  children.  The  number  of  recovered 
children  includes  deceased  children. 

The  Council  provides  training  and  technical  assistance  to  23  “model  courts”  nationwide. 
These  designated  model  courts  seek  to  improve  how  child  abuse  and  neglect  cases  are 
processed  through  the  court  and  child  protection  systems.  Although  their  initiatives  vary, 
all  the  model  courts  address  certain  core  issues,  such  as  how  well  court  processes  work  in 
everyday  practice  and  how  well  the  child  protection  system  meets  the  needs  of  children 
and  their  families. 
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eight  programs  varied,  as  they  were  tailored  to  each  program.  However, 
grantees  in  all  these  programs  were  still  required  to  routinely  report 
narrative  information  on  the  status  of  their  activities  and  accomplishments 
through  semi-annual  progress  reports. 

In  five  of  these  programs,  OJJDP  and/or  an  evaluator  have  established 
specific  reporting  requirements  primarily  to  support  an  outside  program 
evaluation.^®  For  example,  as  a  condition  of  receiving  a  Juvenile  Mentoring 
Program  grant,  OJJDP  requires  all  grantees  to  participate  fully  in  the 
evaluation  by  providing  data  to  the  evaluator.  This  evaluator  requires 
grantees  to  report  their  data  through  quarterly  progress  reports  that  are 
similar  to  semiannual  progress  reports.  The  required  data  include 
information  on  youths  participating  in  each  project,  participating  mentors, 
and  youth-mentor  matches.  For  instance,  grantees  are  required  to  report 
family  structure  information  for  participating  youths.  The  evaluator 
aggregates  such  data  from  all  grantees  and  has  reported,  for  example,  that 
56  percent  of  participating  youths  lived  with  their  mother  only,  20  percent 
lived  with  both  parents,  4  percent  lived  with  their  father  only,  and  21 
percent  were  in  other  hving  arrangements. 

In  two  of  the  eight  programs — ^the  Internet  Crimes  Against  Children  Task 
Force  Program  and  Children’s  Advocacy  Centers — ^specific  reporting 
requirements  were  established  so  that  OJJDP  could  assess  program 
accomplishments.  The  governing  board  of  the  Internet  Crimes  Task  Force, 
in  agreement  with  OJJDP,  identified  monthly  performance  measures  on 
which  grantees  must  report,  such  as  the  number  of  arrests  made,  search 
warrants  issued,  subpoenas  served,  and  cases  opened  by  the  task  forces. 
Under  the  Children’s  Advocacy  Centers  program,"®  OJJDP  prescribed 
specific  performance  measures  on  which  grantees  must  report,  such  as  the 
number  of  practitioners  trained,  training  conferences  held,  and 
publications  distributed.  In  this  program,  specific  reporting  requirements 
were  established  not  only  for  OJJDP  to  assess  the  program’s  overall 
accomplishments,  but  also  to  help  grantees  assess  their  own  projects. 


In  two  of  these  five  programs,  a  secondary  purpose  for  imposing  these  requirements  was 
to  allow  grantees  to  assess  their  own  programs. 

Data  were  reported  in  a  September  2000  OJJDP  Juvenile  Justice  Bulletin  entitled 
Juvenile  Mentoring  Program:  A  Progress  Review. 

This  program  is,  in  part,  a  training  and  technical  assistance  program. 
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The  eighth  program — ^the  Formula  Grants  Program — ^has  requirements  that 
are  statutorily  based  and  further  speUed  out  by  OJJDP  in  program 
regulations.  Program  reporting  requirements  were  established  to  ensure 
grantees  comply  with  the  four  core  requirements  of  the  Juvenile  Justice 
Act  and  as  a  basis  for  assessing  the  effects  of  the  program.  These  core 
requirements  are  (1)  deinstitutionalization  of  status  offenders, (2) 
separation  of  juveniles  from  adult  offenders,  (3)  removal  of  juveniles  from 
adult  jail  and  lockup,  and  (4)  addressing  efforts  to  reduce  disproportionate 
minority  confinement.  OJJDP  regulations  list  in  detail  the  information  on 
which  states  must  report.^®  For  instance,  regarding  the 
deinstitutionalization  of  status  offenders,  states  must  report  the  total 
number  of  accused  and  acUudicated  status  offenders  and  nonoffenders 
placed  in  facilities  that  are,  for  example,  not  near  their  home  community. 
(See  app.  VI  for  a  summary  of  states’  compliance  with  the  core 
requirements  of  the  Juvenile  Justice  Act.)  According  to  the  compliance 
monitoring  coordinator  for  the  Formula  Grants  Program,  grantees’  reports 
on  compliance  with  the  core  requirements  also  provide  the  basis  for 
OJJDP  to  assess  the  effects  of  the  program. 


Some  Data  Exist  on 
the  Number  of 
Juveniles  Served 
Directly,  Although 
Most  Grantees  Are 
Not  Required  to 
Report  This 
Information 


We  identified  eight  programs  that  serve  juveniles  directly  and  whose 
grantees  reported  such  data  for  fiscal  year  2000.  However,  OJJDP  often 
does  not  require  grantees  to  provide  this  information,  in  large  part  because 
not  all  of  its  programs  are  intended  to  provide  direct  services  to  juveniles. 


A  status  offender  is  a  juvenile  who  has  been  charged  with  or  acijudicated  for  conduct  that 
would  not  be  a  crime  if  committed  by  an  adult. 

C.F.R.  31.303(f)(5). 
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In  Some  Programs,  All 
Grantees  Report  the 
Number  of  Juveniles  They 
Directly  Serve 


We  identified  eight  programs  in  which  grantees  directly  serve  juveniles 
and  in  which  all  grantees  report  the  number  of  juveniles  served  to  either 
OJJDP  or  an  outside  evaluator.  About  400  grantees  in  these  eight  programs 
directly  served  close  to  142,000  juveniles  in  one  year.  For  example,  in 
fiscal  year  2000,  the  Juvenile  Mentoring  Program  reported  serving  about 
8,500  juveniles,  and  in  calendar  year  2000,  the  Court  Appointed  Special 
Advocate  Program  reported  serving  70,348  youths.  Table  2  shows  the 
programs  we  identified  as  directly  serving  juveniles  and  reporting  such 
data  for  fiscal  year  2000. 


Table  2:  OJJDP  Programs  GAO  Identified  in  Which  All  Grantees  Reported  Data  on  the  Number  of  Juveniles  Directly  Served, 
Fiscal  Year  2000 

Proqram 

Description 

Juveniles 
directly  served 

Grantees 

Arts  and  At-Risk  Youth 

A  program  that  combines  the  arts  with  job  training  and 
conflict  resolution  skills  and  provides  summer  jobs  or  paid 
internships  for  youths  In  the  proqram. 

381 

3 

The  Arts  Program  for  Juvenile 

Offenders  in  Detention  and  Corrections 

A  program  that  works  with  arts  professionals  to  help  youths 
develop  art  skills  to  use  after  release  from  detention. 

1,026 

6 

Comprehensive  Gang  Initiative 

A  demonstration  program  that  utilizes  a  variety  of  strategies, 
including  intensive  youth  and  family  intervention,  community 
mobilization,  and  organizational  change,  to  reduce  youth 
qanq  crime  and  violence. 

78" 

2 

Court  Appointed  Special  Advocate 
Program 

A  program  to  help  ensure  that  abused  and  neglected  children 
receive  timely  and  effective  representation  in  dependency 
hearings.  Program  volunteers  are  appointed  by  judges  to 
advocate  in  court  for  the  best  interests  of  abused  or 
neqlected  children. 

70,348'’ 

207 

Juvenile  Mentoring  Program  (JUMP) 

A  one-on-one  mentoring  program  for  at-risk  youths. 

8,509 

175 

Life  Skills  Training 

A  training  and  technical  assistance  program  that  instructs 
teachers  on  how  to  replicate  the  Life  Skills  Training  program 
in  a  large  number  of  diverse  jurisdictions.  Life  Skills  Training 
is  a  tested  whole-school-immersion  model  for  reducing  drug 
use  by  middle-school  students  in  selected  communities. 

53,685" 

1 

SafeFutures 

A  program  that  works  to  reduce  delinquency  and  youth 
violence  through  the  development  of  a  continuum  of  services. 
Grantees  in  this  program  use  a  single  grant  to  draw  on  a 
variety  of  OJJDP  categorical  funds. 

5,789" 

6 

Truancy  Reduction  Demonstration 
Program 

A  program  to  reduce  the  number  of  truant  children  by 
providing  services,  such  as  truancy  workshops  and  truancy 
case  workers. 

2,085 

7 

Total 

141,901 

407 

Note:  The  programs  listed  are  those  we  identified  in  which  all  grantees  reported  juveniles-seived 
data.  We  Identified  these  programs  by  reviewing  OJJDP  literature  and  interview/ing  OJJDP  division 
directors.  Nevertheless,  the  list  may  not  be  comprehensive. 
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“The  level  of  services  provided  under  this  program  varied  widely  by  juvenile— i.e.,  some  juveniles 
received  Intensive  inten/ention  counseling,  while  others  were  only  monitored  by  a  case  worker. 

“This  number  represents  juveniles  served  in  calendar  year  2000. 

This  number  Is  associated  with  the  academic  year  1999-2000,  which  roughly  corresponds  with  fiscal 
year  2000.  Because  this  is  a  program  to  teach  drug  prevention  skills  to  all  middle-school  students  in 
selected  communities,  the  grantee  reports  the  number  of  “students  reached”  rather  than  the  number 
of  “juveniles  served.” 

This  number  represents  a  combination  of  data  provided  by  SafeFutures  grantees  and  by  the 
program’s  national  evaluator.  In  addition,  the  level  of  services  provided  to  these  juveniles  varied 
widely  by  individual,  with  some  grantees  including  data  on  juveniles  who  received  a  full  array  of 
project  services,  and  others  reporting  data  on  juveniles  who  were  assessed  but  subsequently  denied 
sen/ices  or  referred  elsewhere.  Furthermore,  one  grantee  reported  data  for  a  slightly  different 
period— September  1999  through  August  2000. 

Source:  GAO  analysis  of  OJJDP,  evaluator,  and  grantee  data. 


Reasons  Exist  for  Not 
Requiring  Grantees  to 
Report  Number  of 
Juveniles  Served 


We  also  identified  a  program  in  which  all  subgrantees  directly  serve 
juveniles,  but  not  all  subgrantees  report  such  data.  The  national  grantee^^ 
for  the  Children’s  Advocacy  Centers  program  reported  that  its  subgrantees 
served  over  100,000  juveniles  in  calendar  year  2000,  but  this  number 
represents  only  those  juveniles  served  by  subgrantees  accredited  through 
a  national  membership  council.^^ 


For  several  reasons,  OJJDP  does  not  typically  require  grantees  to  report 
the  number  of  juveniles  their  projects  directly  serve.  First  and  foremost, 
many  of  OJJDP’s  programs  are  not  intended  to  serve  juveniles  directly. 

The  Juvenile  Justice  Act  established  OJJDP  for  a  variety  of  purposes, 
many  of  which  involve  indirect  benefits  to  juveniles,  rather  than  direct 
services.  Statutorily-established  purposes  for  OJJDP  include  the  following: 

•  To  provide  technical  assistance  to  and  training  programs  for  professionals 
who  work  with  delinquents. 

•  To  provide  for  the  evaluations  of  federally-assisted  juvenile  justice  and 
delinquency  prevention  programs. 

•  To  establish  a  centralized  research  effort  on  problems  of  delinquency. 


The  national  grantee  (one  of  five  program  grantees)  awards  funds  to  subgrantees,  who  in 
turn  provide  direct  services  to  juveniles.  The  other  four  grantees,  which  are  regional, 
provide  training  and  technical  assistance  to  local  Children’s  Advocacy  Centers,  as  does  the 
national  grantee. 

Accredited  subgrantees  received  75  percent  of  funds  available  to  subgrantees. 
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To  assist  state  and  local  governments  in  improving  the  administration  of 
justice  and  services  for  juveniles  who  enter  the  system. 

Some  of  OJJDP’s  programs,  in  their  entirety,  provide  indirect  benefits, 
rather  than  direct  services,  to  juveniles.  OJJDP’s  Model  Courts  Program, 
for  example,  benefits  juveniles  indirectly  by  providing  trainmg  and 
technical  assistance  to  court  personnel  to  improve  their  handling  of  child 
abuse  and  neglect  cases.  The  Internet  Crimes  Against  Children  Task  Force 
Program  also  benefits  juveniles  indirectly  by  helping  to  identify  and  arrest 
pedophiles  and  child  pomographers  who  use  the  Internet  to  prey  on 
children.  Furthermore,  in  commenting  on  a  draft  of  this  report,  the 
Assistant  Attorney  General  pointed  out  that  although  OJJDP’s  research 
projects  do  not  typically  provide  services  directly,  their  results  can 
potentially  help  thousands  of  juveniles. 

OJJDP  officials  provided  the  following  additional  reasons  for  not  requiring 
all  grantees  to  report  the  number  of  juveniles  their  projects  directly  serve; 

A  common  interpretation  of  “juveniles  served”  does  not  exist  across,  or 
even  within,  programs.  For  example,  grantees  in  one  program  might 
consider  the  number  of  juveniles  served  as  those  assessed  for  services  but 
referred  elsewhere,  while  grantees  in  a  different  program  might  consider 
only  juveniles  who  received  at  least  10  sessions  of  therapy.  Even  within 
the  same  program  grantees  may  not  share  a  common  definition  of 
“juveniles  served.”  One  program  grantee  might  report  on  the  number  of 
juveniles  who  received  intensive  one-on-one  drug  prevention  services  over 
an  extended  period  of  time,  while  another  grantee  in  that  same  program 
might  report  on  the  number  who  attended  a  one-time  presentation  on  drug 
prevention.  Without  a  common  interpretation  of  “juveniles  served,”  the 
data  grantees  report  would  be  inconsistent  and  would  have  little  value. 

For  some  programs,  directly  serving  juveniles  may  be  only  one  of  a 
number  of  intended  program  purposes  and  thus,  OJJDP  does  not  typically 
require  all  grantees  within  these  programs  to  report  such  data  For 
example,  in  the  Formula  Grants  Program,  states  and  their  subgrantees  can 
choose  firom  among  14  different  program  areas  related  to  preventing  and 
controlling  delinquency  and  improving  juvenile  justice  systems.  Under  the 
program  area  of  “planning  and  administration,”  for  instance,  states  can 
fund  planning  projects  that  benefit  juveniles  indirectly,  such  as  developing 
a  comprehensive  state  plan  to  identify  juvenile  service  needs  and 
programs  that  address  those  needs  over  the  long  term.  However,  under  the 
area  of  “illegal  drugs  and  alcohol,”  a  local  subgrantee  can  serve  juveniles 
directly  by  establishing  a  drug  and  alcohol  abuse  prevention  project. 
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Methodological 
Concerns  Could 
Adversely  Affect 
Evaluation  Studies 


Juveniles>-seived  data  could  be  used  inappropriately  to  measure  the 
effectiveness  of  a  program  whose  primary  purpose  may  not  be  to  provide 
direct  services  to  juveniles.  For  example,  the  primary  purpose  of  State 
Challenge  Activities  is  to  stimulate  system-wide  change,  although  many  of 
its  10  activity  areas  also  promote  projects  intended  to  directly  serve 
juveniles.  However,  grantees  in  this  program  are  e:5q)ected  to  implement 
direct  service  projects  within  the  broader  context  of  promoting  system- 
wide  change.  For  instance,  one  State  Challenge  Activities  grantee  used 
funds  it  received  under  the  “deinstitutionalization  of  status  offenders” 
activity  area  to  establish  two  community  projects  that  provide  housing  for 
runaway  juveniles,  many  of  whom  are  girls.  The  grantee  intends  to  use  its 
experiences  with  these  two  new  projects  to  initiate  system-wide  change  by 
developing  a  comprehensive  model  program  e3q)ressly  geared  to  serving 
runaway  girls.  By  focusing  on  the  number  of  girls  served  by  this  program, 
one  might  fail  to  see  that  its  primary  purpose  was  to  develop  a 
comprehensive  model  program  for  serving  runaway  girls. 


OJJDP  has  funded  35  evaluations  since  1995,  including  11  evaluations 
intended  to  measure  the  impact  of  OJJDP-funded  programs.  We  reviewed 
the  methodological  rigor  of  10  of  the  11  evaluations."^  Half  of  these  10 
evaluations  are  in  formative  stages,  while  the  other  five  are  weU  into 
implementation."^  None  had  been  completed  at  the  time  of  our  review.  Our 
in-depth  review  of  these  10  evaluations  shows  that  although  several  are 
well-designed  and  use,  or  plan  to  use,  sophisticated  data  analysis  methods, 
others  raise  concerns  as  to  whether  the  evaluations  will  produce  definitive 
results.  We  recognize  that  impact  evaluations,  such  as  the  types  that 
OJJDP  are  funding,  can  encounter  difficult  design  and  implementation 
issues.  For  some  of  the  evaluations  we  reviewed,  program  flexibility  has 
added  to  the  complexity  of  designing  evaluations.  A  lack  of  comparison 
groups  to  aid  in  isolating  the  impacts  of  some  programs,  and  data 
collection  problems  could  compromise  some  evaluation  results. 

According  to  OJJDP  officials,  the  OJJDP  weighs  a  number  of  factors  when 
deciding  which  programs  to  evaluate  and  what  kind  of  evaluations  to  fund. 


""  As  discussed  in  the  Scope  and  Methodology  section  of  this  report,  in  commenting  on  a 
draft  of  this  report,  the  Assistant  Attorney  General  pointed  out  that  the  evalxiation  of  the 
Teen  Courts  program  is  also  an  impact  evaluation  of  an  OJJDP-funded  program.  This 
section  addresses  only  the  10  impact  evaluations  OJJDP  had  initially  identified. 

We  based  our  analysis  of  these  10  ongoing  evaluations  on  documentation  OJJDP 
provided  as  of  i^ril  2001. 


Page  22 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Given  its  budget,  it  considers  how  much  of  its  discretionary  funds  to  spend 
in  support  of  evaluation  activities.  In  deciding  which  of  its  programs  to 
evaluate,  OJJDP  gives  priority  consideration  to  programs  that  have  been 
mandated  by  the  Congress.  Other  criteria  OJJDP  uses  to  determine 
whether  a  program  should  be  evaluated  include  the  program’s  level  of 
funding  and  its  uniqueness,  as  weU  as  the  feasibility  and  cost  of  an 
evaluation  and  its  potential  benefits  to  the  field.  Similar  criteria  are  also 
involved  with  decisions  to  evaluate  non-OJJDP  funded  programs,  as  well 
as  congressional  interest  and  other  federal  agencies’  willingness  to  co-fund 
an  evaluation. 

The  10  impact  evaluations  of  OJJDP-funded  programs  that  we  assessed 
vary  in  size  and  scope.  The  cost  to  conduct  these  evaluations  ranges  firom 
$300,000  to  well  over  $5  million;  however,  some  of  these  grants  involve 
both  impact  and  process  evaluations  and  the  cost  of  the  impact  portion 
alone  cannot  be  separated  from  the  total.  All  10  evaluations  are  multi-year, 
multi-site  projects.  The  number  of  evaluation  sites  ranges  from  2  in  the 
Rural  Gang  Initiative  to  175  in  the  evaluation  of  the  Juvenile  Mentoring 
Program.  As  of  April  2001,  three  evaluations  had  produced  interim  findings 
of  some  program  impacts.^®  (See  app.  n  for  information  on  OJJDP’s 
process  for  disseminating  products  with  interim  findings  as  weU  as  other 
products.) 


Evaluations  of  OJJDP 
Programs  Are  Difficult  to 
Successfully  Design  and 
Implement 


Program  evaluation  is  an  inherently  difficult  task  because  the  objective  is 
to  isolate  the  impact  of  a  particular  program  from  all  other  factors  that 
could  have  caused  a  change  consistent  with  the  intent  of  the  program,  or 
mitigated  against  that  change.  Given  that  programs,  such  as  those  funded 
by  OJJDP,  operate  in  an  ever-changing  environment  and  involve  juveniles 
and  adults  who  themselves  constantly  change,  producing  defimtive 
evaluation  results  can  be  arduous.  For  example,  the  impact  of  a 
hypothetical  program  intended  to  improve  students’  grades  could  be 
confounded  by  the  effects  of  an  outside-of-school  mentoring  program,  the 
transfer  of  high-performing  students  to  a  magnet  program,  changes  in 
school  faculty,  a  new  scholarship  program,  a  severe  flu  season  that  results 
in  widespread  student  absences  from  school,  and  a  myriad  of  other 
factors. 


Two  other  evaluations  reported  preliminary  results  of  some  of  the  data  collected  as  part 
of  the  impact  evaluations;  however,  these  did  not  comprise  impact  results. 
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Project  Variation  Within  a 
Program  Can  Complicate 
Evaluation  Design 


Lack  of  Appropriate 
Comparison  Groups  Could 
Compromise  Some  Results 


Our  in-depth  review  of  the  10  impact  evaluations  of  OJJDP  programs 
showed  that  a  number  of  these  evaluations  are  particularly  complex 
because  local  grantees  design  their  own  projects  to  fit  their  communities’ 
needs.  (See  app.  VII  for  a  description  of  the  impact  evaluations  OJJDP  has 
funded  of  its  own  programs  since  1995).  Although  this  customization  may 
make  sense  from  a  program  perspective,  it  makes  it  more  difficult  to 
evaluate  the  program.  Instead  of  assessing  a  single,  homogeneous  program 
with  multiple  grantees,  the  evaluation  must  assess  the  effects  of  multiple 
configurations  of  a  program.  Although  all  of  the  grantees’  projects  vmder 
each  program  being  evaluated  are  intended  to  achieve  the  same  or  similar 
goals,  an  aggregate  analysis  could  mask  differences  in  individual  projects’ 
effectiveness  and,  thus,  not  result  in  information  about  which 
configurations  of  projects  work  and  which  do  not. 

OJJDP’s  evaluation  of  the  Enforcing  the  Underage  Drinking  Laws  Program 
(discretionary  grant  component)  exemplifies  this  situation.  In 
implementing  their  projects,  states  and  local  communities  have  substantial 
latitude  to  employ  media  campaigns,  merchant  education,  compliance 
checks,  youth  leadership  training,  or  a  variety  of  other  activities  to  deter 
underage  drinking.  Similarly,  under  the  Positive  Action  Through  Holistic 
Education  program,  local  educators  develop  their  own  ways  to  prevent 
student  violence  and  behavior  problems  based  on  their  assessments  of  the 
causes  of  these  problems  in  their  schools.  Because  of  the  limited  number 
of  sites  (two  school  districts)  being  evaluated  and  the  likely  differences  in 
how  each  school  has  developed  its  own  project,  the  resulting  evaluation 
may  not  provide  information  that  could  be  generalized  to  a  broader 
implementation  of  the  program. 

A  standard  way  for  evaluators  to  isolate  the  impacts  of  a  program  from 
other  potential  factors  that  could  have  influenced  change  is  to  use  a 
comparison  group  as  a  benchmark.  In  the  hypothetical  example  cited 
above  concerning  a  program  to  improve  students’  grades,  a  second  set  of 
students  who  are  not  in  the  program  but  are  matched  in  academic 
performance  and  exposed  to  aU  of  the  same  factors  (except  the  program) 
could  provide  a  baseline  from  which  to  assess  the  impact  of  the  program. 
The  grades  of  students  in  the  two  groups  before  and  after  the  program 
would  provide  the  data  from  which  to  measure  program  impacts.  Without 
the  benefit  of  the  comparison  group  as  a  baseline,  it  is  difficult  or 
impossible  to  isolate  changes  resulting  from  the  program  from  changes 
due  to  other  factors. 

The  designs  of  two  of  the  five  evaluations  that  are  well  into 
implementation  lack  an  appropriate  comparison  group.  The  evaluation  of 


Page  24 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Data  Collection  Problems 
Could  Compromise  Some 
Resiilts 


the  Juvenile  Mentoring  Program — a  one-on-one  mentoring  program  for 
youths — compares  youths  entering  the  program  to  those  completing  it. 
However,  a  variety  of  other  factors,  including  the  fact  that  youths  in  the 
program  are  likely  to  mature  and,  thus,  improve  somewhat  spontaneously, 
cannot  be  ruled  out  as  a  rival  cause  of  change  from  the  beginning  to  the 
end  of  the  program.  Although  the  evaluators  are  employing  multiple  and 
innovative  strategies  to  determine  the  effectiveness  of  the  program  in 
achieving  its  objective,  the  lack  of  a  comparison  group  of  nonparticipant 
youths  is  an  obstacle  to  identifying  definitive  outcomes. 

An  evaluation  of  the  Partnerships  to  Reduce  Juvenile  Gun  Violence 
Program  includes  a  comparison  of  before  and  after  crime  statistics  in 
project  communities  with  crime  statistics  for  the  same  time  frames  for  the 
cities  in  which  the  projects  operate.  However,  citywide  crime  statistics 
would  no  doubt  include  data  from  communities  that  are  similar  to  the 
project  community  as  well  as  from  those  that  are  not.  Thus,  the 
differences  between  citywide  and  project  community  baselines  make  it 
difficult  to  attribute  potential  findings  to  the  program. 

Of  the  five  programs  for  which  evaluations  are  still  being  developed,  two 
(the  Safe  Start  Initiative  and  the  Rural  Gang  Initiative)  did  not  seem  to 
have  plans  for  comparison  groups  at  the  time  of  our  review.  Another 
(Parents  Anonymous)  anticipates  using  a  comparison  group,  but  as  yet 
had  not  developed  specific  plans  for  one. 

Regardless  of  the  quality  of  a  program  evaluation  design,  data  collection 
problems  can  compromise  the  validity  of  findings.  Data  collection 
problems  may  affect  the  validity  of  the  findings  for  three  of  the  five 
evaluations  that  are  currently  completing  or  have  completed  data 
collection.  The  Juvenile  Mentoring  Program  evaluation  has  experienced 
problems  obtaining  behavioral  measures  and  school  performance  data 
with  which  to  gauge  program-driven  change.  The  Comprehensive  Gang 
Initiative  evaluation  has  also  experienced  data  collection  problems  such  as 
the  lack  of  fully  adequate  comparison  youth  data  at  all  or  most  sites, 
missing  police  histories,  and  missing  self-reported  data. 

The  Intensive  Aftercare  evaluation  has  e3q)erienced  survey  response  rate 
shortfalls,  in  some  cases  obtaining  response  rates  of  less  than  30  percent, 
which  may  affect  the  validity  of  the  findings.  In  commenting  on  a  draft  of 
this  report,  the  Assistant  Attorney  General  said  that  the  poor  response 
rates  for  some  elements  at  different  sites  were  particularly  disappointing 
because  this  evaluation  had  a  strong  random  assignment  design;  however, 
the  strategies  for  obtaining  adequate  data  turned  out  to  be  insufficient.  She 


Page  25 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


added  that  the  program  staff  who  were  required  to  collect  data  did  not 
give  data  collection  adequate  priority  in  comparison  to  their  other  duties. 
This  was  particularly  true  of  data  regarding  the  comparison  groups. 


OJJDP  Has  Funded 
Other  TVpes  of 
Evaluations 


In  addition  to  funding  impact  evaluations  of  OJJDP  programs,  OJJDP  has 
funded  24  other  evaluations  since  1995 — 11  nonimpact  evaluations  of 
OJJDP  programs,  9  impact  evaluations  of  programs  that  were  not 
supported  by  OJJDP  funds,  and  4  nonimpact  evaluations  of  programs  that 
were  not  funded  by  OJJDP.  The  nonimpact  evaluations  are  not  intended  to 
determine  the  outcomes  of  the  various  programs,  but  rather  how  well  the 
programs  have  been  implemented.  For  example,  OJJDP  has  funded  a 
process  evaluation  of  its  SafeFutures  program  to  learn  more  about  the 
process  of  community  mobilization  and  collaboration  in  building  a 
comprehensive  program  of  prevention  and  intervention  for  at-risk  youths 
and  juvenile  offenders. 

OJJDP  has  also  funded  evaluations  of  programs  that  are  funded  by  entities 
other  than  OJJDP.  For  example,  although  OJJDP  does  not  fund  the  Act 
Now  Truancy  Program,  it  has  funded  a  nonimpact  evaluation  of  this 
program.  The  Act  Now  Truancy  Program  grew  out  of  a  unique  Arizona  law 
that  allowed  prosecutors  to  issue  citations  to  parents  whose  children  were 
chronically  truant.  Because  there  was  a  great  deal  of  interest  in  this 
approach  and  OJJDP  believed  it  provided  a  unique  opportunity  to  learn 
about  the  impact  of  an  unusual  approach,  it  funded  an  evaluation  of  the 
program.  Appendix  VIII  contains  brief  descriptions  of  these  24  other 
evaluations. 


Conclusions 


Although  there  is  great  interest  in  assessing  results  of  programs,  it  is 
extremely  difficult  to  design  and  execute  evaluations  that  will  provide 
definitive  information.  Our  in-depth  review  of  10  OJJDP-ftinded 
evaluations  of  OJJDP's  own  programs  undertaken  since  1995  has  shown 
that,  in  some  cases,  the  flexibility  that  can  be  beneficial  to  grantees  in 
tailoring  programs  to  meet  their  communities'  needs  has  added  to  the 
complexities  of  designing  impact  evaluations  that  will  result  in  valid 
findings.  Furthermore,  the  lack  of  an  appropriate  comparison  group  or 
sites  and/or  problems  in  data  collection  may  compromise  the  reliability 
and  validity  of  some  of  these  evaluations.  Because  half  of  these  10 
evaluations  are  in  relatively  early  stages,  any  potential  problems  with 
comparison  group  issues  or  data  collection  shortfalls  could  still  be 
resolved  over  the  course  of  the  evaluation.  We  recognize  that  not  all 
evaluation  issues  that  can  compromise  results  are  resolvable,  including 
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many  involving  comparison  groups  and  data  collection.  However,  to  the 
extent  that  appropriate  comparison  groups  can  be  established  and  tracked 
and  data  collection  issues  can  be  overcome,  the  validity  of  the  evaluation 
results  can  be  enhanced. 


R#ir*r»mTnpnHjitinn  fnr  review  of  the  recent  OJJDP  program  evaluations  has  shown  that,  of 

ivecul  1 11 1  lei  lUdtiui  i  iui  ^  stages,  some  problems  with  valid 

Ex6CUtiV6  Action  comparison  groups  and/or  data  collection  could  compromise  the 

usefulness  of  some  of  their  results.  Five  other  program  evaluations  are  in  a 
formative  stage  where  comparison  group  issues  and  data  collection 
strategies  are  not  yet  finalized.  Accordingly,  we  recommend  that  the 
Attorney  General  direct  the  Administrator  of  OJJDP  to  assess  the  five 
impact  evaluations  in  the  formative  stages  to  address  potential 
comparison  group  and  data  collection  problems  and,  on  the  basis  of  that 
assessment,  initiate  any  needed  interventions  to  help  ensure  that  the 
evaluations  produce  definitive  results. 


Agency  Comments 
and  Our  Evaluation 


We  provided  a  copy  of  this  report  to  the  Attorney  General  for  review  and 
comment.  In  an  October  15,  2001  letter,  the  Assistant  Attorney  General 
commented  on  a  draft  of  this  report.  Her  comments  are  summarized  below 
and  are  presented  in  their  entirety  in  appendix  IX.  Her  detailed  comments 
have  been  addressed  in  the  report  as  appropriate. 

The  Assistant  Attorney  General  said  that  the  draft  report  provides  useful 
information  that  highlights  areas  warranting  attention.  She  added  that  the 
draft  report  would  be  an  important  tool  that  OJP  will  use  to  improve  the 
quality  of  its  evaluations  and  to  design  programs  that  wiU  achieve  greater 
impact.  Furthermore,  OJP  will  assess  the  five  impact  evaluations  that  are 
currently  in  their  formative  stages  to  address  potential  comparison  group 
and  data  collection  problems.  On  the  basis  of  that  assessment,  OJJDP  wiU 
initiate  any  needed  interventions  to  help  ensure  that  evaluations  produce 
definitive  results. 

The  Assistant  Attorney  General  said  that  OJP  agrees  that  it  should  always 
strive  for  more  rigorous  and  scientifically  sound  evaluation  designs  and 
that  the  inclusion  of  comparison  groups  would  certainly  strengthen  the 
interpretation  of  evaluation  results.  However,  she  disagreed  with  our 
reliance  on  the  use  of  comparison  groups  as  the  only  valid  evaluation 
design  for  two  primary  reasons.  First,  OJJDP  seeks  to  conduct  juvenile 
justice  evaluations  in  a  real-world  setting,  where  laboratory-like 
comparison  groups  may  not  be  possible.  Second,  sufficient  funding  is  not 
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available  for  including  comparison  groups  in  every  evaluation.  The 
Assistant  Attorney  General  also  said  that  given  the  choice  between 
conducting  far  fewer  evaluations,  all  with  comparison  groups,  and 
conducting  a  greater  number  of  evaluations  under  less-than-ideal 
conditions,  OJJDPs  Research  and  Program  Development  Division  works 
hard  to  tread  a  middle  ground  that  satisfies  needs  for  both  quality  and 
quantity.  She  further  pointed  out  that  a  growing  number  of  policy  makers 
and  evaluators  firmly  believe  that  community-based  initiatives  do  not  lend 
themselves  to  the  kind  of  traditional  evaluations  that  this  draft  report 
proposes.  Accordingly,  some  researchers  have  strongly  urged  that  new 
approaches  to  evaluation  be  developed. 

In  addition,  the  Assistant  Attorney  General  said  that  our  report  suggests 
that  more  evaluations  using  experimental  or  quasi-experimental  evaluation 
designs  should  be  funded.  She  added  that  many  communities  reject 
participation  in  programs  that  are  evaluated  in  this  way  (i.e.,  with  control 
or  comparison  groups)  because  they  feel  that  it  requires  them  to  purposely 
exclude  youths  from  receiving  services. 

In  her  comments,  the  Assistant  Attorney  General  seemed  to  be  using  the 
terms  “comparison  group”  and  “control  group”  interchangeably.  However, 
control  groups  are  commonly  associated  with  experiments  involving 
random  assignment.  We  do  not  intend  our  statements  regarding  the  need 
for  comparison  groups  in  impact  evaluations  to  imply  that  random 
assignment  is  necessary  for  studies  to  be  valid.  Furthermore,  we  recogmze 
that  groups  can  be  compared  after  controlling  for  differences  by  methods 
other  than  random  assignment,  including  statistical  methods  and  various 
methods  of  matching.  For  impact  evaluations,  comparisons  should  be 
made,  and  should  involve  individuals  who  were  not  subject  to  the  program 
or  treatment  bemg  evaluated.  However,  not  aU  the  evaluations  we 
assessed  made  such  comparisons. 

We  also  recognize  that  not  all  evaluation  issues  that  can  compromise 
results  are  resolvable,  even  with  the  use  of  comparison  groups.  We  also 
recognize  that  designing  evaluations  with  comparison  groups  can  be 
expensive  and  funding  limitations  could  preclude  their  use  in  all 
evaluations.  In  addition,  obtaining  participants  can  be  troublesome,  as  the 
Assistant  Attorney  General  pointed  out.  However,  the  validity  of 
evaluation  results  can  be  enhanced  through  establishing  and  tracking 
comparison  groups.  If  other  ways  exist  to  effectively  isolate  the  impacts  of 
a  program,  comparison  groups  may  not  be  needed.  However,  we  saw  no 
evidence  of  other  methods  being  used  in  the  10  impact  evaluations  we 
assessed.  While  studies  that  do  not  have  appropriate  comparison  groups 
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can  provide  useful  information,  they  should  not  be  considered  impact 
evaluations.  Furthermore,  we  recognize  the  fact  that  communities  may  not 
favor  withholding  treatments  or  programs  from  individuals  in  control  or 
comparison  groups,  however,  this  problem  is  commonly  handled  by 
phasing  in  the  treatment  or  program  and  offering  it  to  comparison  group 
members  following  the  evaluation  period. 


As  we  agreed  with  your  office,  unless  you  publicly  announce  the  contents 
of  this  report  earlier,  we  plan  no  further  distribution  of  it  xmtil  30  days 
from  the  date  of  this  letter.  At  that  time,  we  will  send  copies  to  the  Senate 
Judiciary  Committee,  the  Senate  Subcommittee  on  Youth  Violence,  the 
House  Committee  on  Education  and  the  Workforce,  the  House 
Subcommittee  on  Early  Childhood,  Youth  and  Families,  the  Attorney 
General,  and  the  Director  of  the  Office  of  Management  and  Budget. 

If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
James  M.  Blume  or  me  at  (202)  512-8777.  Key  contributors  to  this  report 
are  acknowledged  in  appendix  X. 

Sincerely  yours, 


Laurie  E.  Ekstrand 
Director,  Justice  Issues 
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Appendix  I:  OJJDP  Awards  Data,  Fiscal  Years 
1996  Through  2000 


This  appendix  provides  information  on  the  awards  the  Office  of  Juvenile 
Justice  and  Delinquency  Prevention  (OJJDP)  made  each  year  from  fiscal 
years  1996  through  2000.  It  contains  data  on  OJJDP  funds  awarded  to 
formula/block  grant  programs  versus  discretionary  grant  programs  (see 
fig.  2),  OJJDP  funds  awarded  by  more  specific  program  areas  (see  table  3), 
types  of  OJJDP  award  recipients  (see  table  4),  and  OJJDP  formula/block 
grant  awards  by  state  (see  table  5). 

We  relied  on  the  Office  of  Justice  Program’s  (OJP)  awards  database  to 
analyze  data  on  all  OJJDP-administered  awards  made  during  this  5-year 
period.  We  analyzed  awards  by  the  year  the  award  was  made — ^not  the 
year  in  which  the  funds  were  appropriated.  We  worked  with  OJJDP 
officials  to  identify  awards  by  m^gor  program  or  program  area,  as  the 
database  did  not  provide  sufficiently  detailed  information.  OJP  officials 
advised  us  that  they  perform  daily  quality  control  checks  on  all  data 
entered  into  their  database,  however,  we  did  not  verify  the  accuracy  of  the 
database. 


Figure  2:  OJJDP  Formula/Block  Grants  Versus  Discretionary  Grant  Awards,  Fiscal 
Years  1996  Through  2000 
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Note:  Dollar  amounts  are  rounded  to  the  nearest  million. 
Source:  GAO  analysis  of  OJP  database. 
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Appendix  I:  O JJDP  Awards  Data»  Fiscal  Years 
1996  Through  2000 


Table  3:  OJJDP  Funds  Awarded  by  Program  Area,  Fiscal  Years  1996  Through  2000 


Proqram  area 

FY1996 

FY1997 

FY1998 

FY1999 

FY2000 

Formula/block  grant  programs 

Formula  Grants  Program 

$68,795,296 

$77,469,250 

$92,069,600 

$74,468,599 

$69,699,750 

State  Challenge  Activities 

9,352,900 

10,363,264 

9,726,800 

9,011,429 

8,227,500 

Community  Prevention  Grants  Program 
(Title  V) 

19,762,000 

18,368,000 

18,707,000 

39,921,000 

36,158,000 

Enforcing  the  Underage  Drinking  Laws 
Proqram  (block  grant  component) 

a 

18,360,000 

17,640,000 

19,080,000 

Juvenile  Accountability  Incentive  Block 

Grants  Proqram 

a 

a 

232,250,000 

232,250,000 

221.094,775 

Discretionary  grant  programs 

National  Institute  for  Juvenile  Justice  and  Delinquency  Prevention*’ 

Comprehensive  Strategy  for  Serious, 

Violent,  and  Chronic  Juvenile  Offenders 

900,730 

900,055 

1,835.597 

1,842,240 

2,640,000 

Causes  and  Correlates  of  Delinquency 
Research 

299,999 

900,000 

599,999 

600,000 

750,000 

Statistics  and  Systems  Development 

Program 

349,978 

249,996 

449,998 

549,923 

649,996 

Other  research,  statistics,  evaluation, 
information  dissemination,  and  training  and 
technical  assistance  grants 

12,382,088 

15,689,329 

21 ,566,738 

40,816,294 

42,783,923 

Demonstration  and  replication  programs 

Comprehensive  Gang  Initiative 

2,349,597 

2,848,881 

3,224,480 

2,249,742 

1,149,498 

Rural  Gang  Initiative 

a 

a 

215,000 

1,112,754 

1,474,473 

SafeFutures 

2,159,950 

5,540,000 

4,715,979 

9,462,844 

10,829,102 

Community  Assessment  Centers 

a 

1,074,552 

1,699,992 

125,000 

875,000 

Juvenile  Mentoring  Program  (JUMP) 

a 

9,688,488 

3,363,281 

21,059,220 

5,760,000 

Partnerships  to  Reduce  Juvenile  Gun 

Violence  Program 

a 

551 ,060 

1,050,230 

1,299,975 

799,996 

Enforcing  the  Underage  Drinking  Laws 
Program  (discretionary  grant  component) 

a 

a 

6,608,019 

2,344,986 

6,272,003 

Drug-Free  Communities  Support  Program 

a 

a 

8,903,433 

19,039,326 

29,677,581 

Truancy  Reduction  Demonstration  Program 

a 

a 

693,849 

711,328 

Tribal  Youth  Program 

a 

a 

a 

a 

8,252,019 

Other  demonstration  and  replication  grants 

7,070,330 

8,846,421 

17,178,085 

26,309,408 

24,058,149 

Missing  and  Expioited  Chiidren  Program 

National  Clearinghouse  and  Resource 

Center  for  Missing  and  Exploited  Children 

3,294,648 

4,576,640 

7,585,000 

8,925,000 

19,490.263 

Missing  and  Exploited  Children  Training  & 
Technical  Assistance  program 

a 

3,448,575 

1,749,803 

1,848,947 

1,949,741 

Internet  Crimes  Against  Children  Task  Force 
Program 

a 

a 

2,351 ,829 

199,997 

8,486,056 

Other  missing  and  exploited  children  grants 

1,189,639 

1,488,935 

1,759,887 

978,356 

2,074,318 

Child  Abuse  and  Neqlect  Proqram 

Court  Appointed  Special  Advocate  Program 

6,040,000 

5,904,400 

7,062,100 

9,034,200 

9,796,000 

Children’s  Advocacy  Centers 

3,199,856 

3,099,914 

5,691,000 

4,733,000 

5,824,302 
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Program  area 

FY1996 

FY1997 

FY1998 

FY1999 

FY2000 

Model  Courts  Program 

742,500 

984,100 

1,964,900 

2,108,097 

2,109,200 

Safe  Start  Initiative 

a 

a 

a 

a 

8,038,573 

Other  child  abuse  and  neglect  grants 

1,725,000 

3,199,878 

4,838,152 

6,508,938 

6,499,976 

Total 

$139,614,511 

$175,191,738 

$475,526,902 

$535,133,124 

$555,211,522 

Note:  Award  amounts  are  not  adjusted  for  de-obligated  funds,  except  in  cases  in  which  an  entire 
award  was  de-obligated  and  later  re-awarded,  as  occurred  in  rare  instances  of  formula/bock  grant 
awards. 


Tither  the  program  did  not  receive  an  appropriation  or  OJJDP  did  not  award  funds. 

‘’Awards  listed  under  this  subheading  comprise  a  variety  of  research,  statistics,  evaluation, 
information  dissemination,  and  training  and  technical  grants.  However,  the  funds  awarded  under  this 
subheading  do  not  represent  a  comprehensive  accounting  of  all  such  activities  funded  by  CXJJDP.  In 
some  cases  these  activities  are  accounted  for  under  other  subheadings.  For  example,  the  Missing 
and  Exploited  Children  Training  and  Technical  Assistance  award,  which  is  primarily  a  training  and 
technical  assistance  grant,  is  listed  under  the  Missing  and  Exploited  Children’s  Program.  Also, 
evaluations  for  many  of  the  demonstration  and  replication  programs  are  accounted  for  under  their 
respective  programs,  rather  than  under  the  National  Institute  for  Juvenile  Justice  and  Delinquency 
Prevention.  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  pointed  out  that 
combining  training  and  technical  assistance  funding  under  the  National  Institute  of  Justice  and 
Delinquency  Prevention  does  not  highlight  the  importance  of  training  and  technical  assistance. 
Although  we  recognize  the  Importance  of  training  and  technical  assistance,  OJP’s  awards  database 
did  not  readily  lend  itself  to  identifying  all  training  and  technical  assistance— either  awards  or  portions 
of  awards— because  they  are  part  of  many  programs  across  OJJDP  divisions. 

Source:  GAO  analysis  of  OJP  database. 


Table  4:  OJJDP  Award  Recipients,  Fiscal  Years  1996  Through  2000 

FY1996 

FY1997 

FY1998 

FY1999 

FY2000 

Recipients 

Amount 

awarded 

%“ 

Amount 

awarded 

%* 

Amount 

awarded 

% 

Amount 

awarded 

%* 

Amount 

Awarded 

%“ 

State  governments 

$98,854,634 

71 

$109,350,562 

62 

$377,103,675 

79 

$377,819,401 

71 

$369,196,062 

67 

Local  governments^ 

2,584,003 

2 

7,230,243 

4 

13,534,664 

3 

18,766,350 

4 

27,078,615 

5 

Schools'" 

4,936,373 

4 

14,425,127 

8 

18,292,717 

4 

20,926,891 

4 

34,975,373 

6 

Nonprofit  organizations 

32,289,708 

23 

41,026,686 

23 

60,476,935 

13 

106,044,793 

20 

109,431,488 

20 

other" 

949,793 

1 

3,159,120 

2 

6,118,911 

1 

11,575,689 

2 

14,529,984 

3 

Total 

$139,614,511 

100 

$175,191,738 

100 

$475,526,902 

100 

$535,133,124 

100 

$555,211,522 

100 

“Percentages  may  not  total  100  due  to  rounding. 


“Local  governments  include  county  governments,  cities,  townships,  and  special  district  governments. 
‘’Schools  Include  Independent  school  districts  and  state  and  private  institutions  of  higher  education. 
“Other  includes  for-profit  organizations,  individuals,  Indian  tribes,  and  other  miscellaneous  categories. 
Source:  GAO  analysis  of  OJP  database. 
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Table  5:  OJJDP  Formula/Block  Grant  Awards,  by  State,  Fiscal  Years  1996  Through  2000 


Funds  awarded 


state 

FY1996 

FY1997 

FY1998 

FY1999 

FY2000 

Alabama 

$1,424,000 

$1,648,900 

$5,896,400 

$5,855,300 

$5,638,100 

Alaska 

787,500 

832,500 

2,822,300 

2,839,800 

2,594,200 

Arizona 

1,502,000 

1,821 ,000 

6,197,500 

6,714,900 

6,327,800 

Arkansas 

869,000 

1,014,500 

4,222,700 

4,432,700 

4,222,100 

California 

11,440,000 

13,417,100 

37,575,000 

37,547,300 

36,119,800 

Colorado 

1,279,000 

1,497,000 

5,579,400 

5,649,200 

5,474,219 

Connecticut 

1,017,000 

1,197,000 

4,744,200 

4,824,300 

4,612,400 

Delaware 

787,500 

829,500 

2,133,100 

3,469,600 

2,726,200 

District  of  Columbia 

788,546 

187,500 

1,785,400 

1,778,000 

1,699,100 

Florida 

4,302,000 

5,144,000 

15,440,600 

15,631,200 

15,089,600 

Georgia 

2,467,000 

2,907,000 

9,432,800 

9,534,600 

9,188,600 

Hawaii 

787,500 

860,500 

2,984,800 

3,402,400 

3,096,800 

Idaho 

787,500 

869,500 

3,280,000 

3,388,200 

3,264,700 

Illinois 

3,605,000 

1,626,064 

17,824,400 

14,320,800 

12,886,850 

Indiana 

1,942,000 

2,269,000 

7,615,300 

7,608,129 

7,304,900 

Iowa 

961,000 

1,106,000 

4,446,700 

4,641,700 

4,438,300 

Kansas 

909,000 

1,057,000 

4,316,400 

4,517,200 

4,360,300 

Kentucky 

278,000 

0 

4,117,800 

5,117,100 

6,659,600 

Louisiana 

1,553,900 

1,816,100 

6,518,200 

6,317,100 

6,010,400 

Maine 

787,500 

859,500 

2,965,650 

3,027,850 

3,076,500 

Maryland 

1,665,000 

1,941 ,000 

6,751,400 

6,678,100 

6,418,700 

Massachusetts 

1,537,500 

893,000 

7,789,450 

7,375,900 

7,035,600 

Michigan 

5,572,000 

3,808,000 

11,800,200 

11,613,500 

11,207,319 

Minnesota 

1,636,000 

1,899,000 

6,592,150 

6,604,300 

6,312,800 

Mississippi 

997,000 

857,000 

4,902,400 

4,736,500 

4,327,600 

Missouri 

1,797,000 

2,088,000 

7,169,800 

7,188,300 

6,835,700 

Montana 

787,500 

843,500 

2,957,900 

2,619,400 

3,223,100 

Nebraska 

664,500 

910,500 

3,558,900 

3,700,700 

3,563,000 

Nevada 

795,500 

888,500 

3,481,600 

3,695,300 

3.630,700 

New  Hampshire 

637,500 

1,006,500 

3,133,550 

3,192,800 

3,089,400 

New  Jersey 

2,546,000 

2,996,000 

7,096,000 

7,676,900 

9,528,200 

New  Mexico 

830,500 

939,500 

3,739,300 

3,884,500 

3,748,900 

New  York 

5,815,000 

6,798,000 

19,858,900 

19,779,400 

18,544,600 

North  Carolina 

2,260,000 

2,693,000 

8,924,300 

9,016,600 

8,746,900 

North  Dakota 

787,500 

827,500 

2,779,400 

2,766,400 

2,682,900 

Ohio 

3,763,000 

4,363,000 

13,101 ,700 

13,060,100 

9,405,700 

Oklahoma 

1,160,000 

1,340,000 

5,102.900 

5.170,600 

4,946,500 

Oregon 

1,027,000 

1,217,000 

4,809,400 

4,934,700 

3,769,500 

Pennsylvania 

3,821,000 

4,439,000 

13.292,600 

13,163,000 

12,471,600 

Rhode  Island 

787,500 

843,500 

2,965,000 

2,992,300 

2,905,700 
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Funds  awarded 


state 

FY1996 

FY1997 

FY1998 

FY1999 

FY2000 

South  Carolina 

1,240,000 

1,427,000 

5,319,300 

5,166,300 

4,938,550 

South  Dakota 

787,500 

100,000 

2,013,500 

1,993,800 

1,921,300 

Tennessee 

1,710,000 

1,999,000 

6,897,100 

6,905,800 

5,332,000 

Texas 

6,910,000 

8,165,000 

23,614,200 

24,006,200 

22,961,500 

Utah 

885,000 

1,030,000 

4,281,400 

4,518,500 

4,372,000 

Vermont 

787,500 

822,500 

2,717,300 

2,711,200 

2,626,619 

Virginia 

2,113,000 

2,460,000 

8,155,800 

8,205,799 

7,794,800 

Washington 

1,856,000 

2,164,000 

7,362,500 

7,433,800 

6,784,350 

West  Virginia 

810,500 

899,500 

3,497,100 

3,590,800 

3.432,100 

Wisconsin 

1,755,000 

2,045,000 

6,538,150 

7,368,750 

6,621,600 

Wyoming 

0 

0 

1,933,950 

1,836,800 

1,759,100 

American  Samoa 

148,000 

133,000 

617,391 

601,962 

472,248 

Guam 

115,000 

192,000 

842,350 

841,700 

807,800 

Northern  Mariana  Islands 

148,000 

151,000 

381,959 

381,738 

370,552 

Puerto  Rico 

1,367,750 

1,917,000 

5,856,900 

5,854,100 

5,548,800 

Republic  of  Pulau 

15,000 

21,350 

4,300 

0 

0 

Virgin  Islands 

100,000 

123,000 

1,374,700 

1,407,100 

1,331,818 

Total 

$97,910,196 

$106,200,514 

$371,113,400 

$373,291,028 

$354,260,025 

Source:  GAO  analysis  of  OJP  database. 
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The  Office  of  Juvenile  Justice  and  Delinquency  Prevention  (OJJDP)  has  a 
process  for  disseminating  published  interim  results  of  impact  evaluations 
as  weU  as  other  publications  produced  by  OJJDP  and  its  grantees.  OJJDP 
publications  are  available  through  the  Juvenile  Justice  Clearinghouse.^ 
According  to  an  OJJDP  official,  OJJDP  develops  a  specific  strategy  for 
each  publication  that  includes  the  number  of  copies  to  be  printed,  the 
methods  for  announcing  availability,  and  the  target  audience  that  will 
automatically  receive  copies.  OJJDP  promotes  products  through  the 
National  Criminal  Jnstice  Reference  Service  (NCJRS)  Catalog,  OJJDP’s 
Juvenile  Justice ']owxdl,  the  NCJRS  and  OJJDP  Web  sites,  e-mail  lists,  the 
Office  of  Justice  Programs  press  annoxmcements,  conference  displays, 
criminal/juvenile  justice  newsletters  and  journals,  and  flier  mailings. 
Almost  all  of  OJJDP’s  publications  are  made  available  to  the  public 
through  OJJDP’s  Web  sites,  which  is  administered  by  the  Juvenile  Justice 
Clearinghouse.  Many  publications,  depending  on  their  length,  are  also 
available  through  the  Clearinghouse’s  fax-on-demand  service.  Individuals 
can  also  order  copies  of  publications  online  or  by  calling  the 
Clearinghouse’s  toll-free  number.  In  addition,  the  Clearinghouse 
automatically  sends  publications  to  targeted  constituents  (e.g.,  juvenile 
justice  policymakers,  practitioners,  researchers,  and  community-based 
organizations)  and  to  individuals  who  have  registered  to  receive 
publications  based  on  their  specific  areas  of  interest. 

As  of  May  2001,  OJJDP  had  used  this  dissemination  process  to  share 
interim  results  from  5  of  the  10  ongoing  impact  evaluations^  of  OJJDP 
programs  that  we  assessed.^  In  total,  OJJDP  had  distributed  over  400,000 
copies  of  9  products  that  contained  interim  results  from  the  5  evaluations. 
Table  6  provides  additional  information  on  the  distribution  of  these 
publications. 


^  OJJDP  established  the  Juvenile  Justice  Clearinghouse  to  provide  individuals  and 
organizations  easy  access  to  information  and  resources  on  juvenile  justice  topics.  The 
Clearinghouse  is  a  component  of  NCJRS,  a  federally-sponsored  information  clearinghouse 
that  offers  a  range  of  services  including  online  newsletters,  access  to  publications  by 
partner  agencies,  and  a  calendar  of  upcoming  conferences.  Its  federal  sponsors  include  the 
Office  of  Justice  Programs  (including  OJJDP)  and  the  Office  of  National  Drug  Control 
Policy.  NCJRS  maintains  a  Web  site  where  publications  by  OJJDP  and  other  federal 
sponsors  can  be  accessed, 

^  Although  these  interim  results  were  based  on  data  collected  from  impact  evaluations,  they 
did  not  necessarily  comprise  impact  results. 

^  These  10  impact  evaluations  were  ongoing  as  of  April  2001. 
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Table  6:  Dissemination  of  Published  Products  With  Interim  Results  of  Impact  Evaluations,  as  of  May  2001 

Program 

Product* 

Automatic 

mailing*' 

other 

distribution* 

other 

availability 

Juvenile  Mentoring  Program 
(JUMP) 

Juvenile  Mentoring  Program:  1998  Report  to  the 
Congress  (Report,  December  1 998). 

41,000 

7,100 

Online 

Juvenile  Mentoring  Program:  A  Progress  Review 
(Bulletin,  September  2000). 

44,000 

7,000 

Online 

Fax-on-demand 

FY  2000  OJJDP  Discretionary  Program 
Announcement:  Juvenile  Mentoring  Program 
(Solicitation,  March  2000). 

63,700 

9,000 

Online 

Fax-on-demand 

Comprehensive  Gang 

Initiative 

Gang-Free  Schools  and  Communities  Initiative:  FY 
2000  OJJDP  Discretionary  Program  Announcement 
(Solicitation,  July  2000). 

48,700 

6,700 

Online 

Fax-on-demand 

Youth  Gang  Program  and  Strategies  (Summary, 
August  2000). 

45,000 

2,500 

Online 

intensive  Aftercare 

Implementation  of  the  Intensive  Community-Based 
Aftercare  Program  (Bulletin,  July  2000). 

32,100 

2,500 

Online 

Fax-on-demand 

Reintegrating  Juvenile  (Offenders  Into  the 

Community:  OJJDP’s  Intensive  Community-Based 
Aftercare  Demonstration  Program  (National  Institute 
of  Justice  Research  Preview,  December  1 998). 

Not 

available 

664 

Online 

Fax-on-demand 

Partnerships  to  Reduce 
Juvenile  Gun  Violence 
Program 

Fighting  Juvenile  Gun  Violence  (Bulletin,  September 
2000). 

44,000 

2,300 

Online 

Fax-on-demand 

Enforcing  the  Underage 
Drinking  Laws  Program 

OJJDP  Research  2000  (Report,  May  2001). 

46,000 

1,366 

Online 

Total 

364,500 

39,130 

Tull  citation  information  for  these  products  is  contained  in  the  bibliography. 


'^‘Automatic  mailing”  refers  to  publications  disseminated  to  targeted  and  registered  individuals  and 
groups.  The  Juvenile  Justice  Clearinghouse  maintains  290  targeted  mailing  lists  of  key  constituents 
that  include  criminal/juvenile  justice  policymakers,  practitioners,  researchers,  and  community-based 
organizations.  In  addition,  individuals  can  register  with  the  Clearinghouse  to  automatically  receive 
information  based  on  their  specific  areas  of  interest. 

‘'“Other  distribution”  refers  to  orders  for  specific  publications.  Anyone  can  order  a  publication  by  calling 
the  toll-free  line  or  using  the  NCJRS  or  OJJDP  Web  site. 

Source:  OJJDP. 
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U.S.  DKPARTMENT  OF  JCSTICE 
Office  of  JaMke  PrQfrani!^ 


CATEGORICAL  ASSISTANCE  PROGRESS  REPORT 


Tile  mfonnatlon  provided  will  be  used  by  the  grantor  agency  lo  monitor  grantee  cash  flow  to  entire  proper  use  of  Federal 
funtb.  No  fttcth^  rnooks  «r  other  tenefits  may  be  paid  out  under  this  program  unless  this  report  is  completed  and  filed  as 
required  by  emsting  law  and  regulations  (Uidform  Adminisiraii\'e  Requirements  for  Grants  and  Cooperatiw  Agreements  — 
2S  CFR,  Part  66,  Common  Rule,  and  0MB  Circular  A>l  I0|. 
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INSTRUCTIONS  FOR  COMPLETING  OJP  CATEGORICAL  ASSISTANCE  PROGRESS  REPORTS 

Grantees  are  required  to  submit  Progress  Reports  on  project  activities  and  accomplishments.  It  is 
expected  that  reports  will  include  data  appropriate  to  this  stage  of  project  development  and  in  sufficient 
detail  to  provide  a  clear  idea  and  summa^  of  work  and  accomplishmenis  to  date.  The  following  should 
be  observed  in  preparation  and  submission  of  progress  reports: 

a-  Due  Date.  Reports  are  due  on  a  semi<annual  basis  as  follows 

Reporting  Period:  Due  Not  Later  Than: 

Jan1^une30  July  30 

July1-E>ec31  Jan  30 

A  final  report  is  due  90  days  following  the  close  of  the  grant  period  or  any  extension  thereof. 

b.  Submission .  Grantees  shall  submit  to  OJP  Office  of  the  Comptroller  an  original  and  one  copy 
of  quarterly  progress  reports  and  an  original  and  three  copies  of  final  progress  reports.  Reports 
should  be  mailed  to: 

Office  of  Justice  Programs 
Office  of  the  Comptroller  /  Control  Desk 
810  Seventh  Street,  NW 
Washington,  DC  20531 

c.  Form  and  Execution  .  Grantees  should  use  OJP  Form  4587/1  as  a  fece  sheet  If  continuation 
pages  are  needed,  plain  bond  paper  is  to  be  used.  The  report  is  to  be  signed  by  the  person 
designated  as  project  director  on  the  grant  application  or  any  duly  designated  successor. 

d.  Repoitinq  Reauirsments  .  The  reporting  requirements  are  intended  to  provide  the  grantor 
agency  with  sufficient  information  to  monitor  grant  implementation  and  goal  achievement.  To 
support  effective  monitoring,  progress  reports  must  be  keyed  to  the  grant  implementation  plan 
provided  in  Part  IV  of  the  grant  application.  Specifically,  the  report  should: 

1 .  Indicate  the  status  of  each  goal  which  was  due  for  completing  during  a  previous  report 
quarter  but  carried  over  due  to  implementation  or  other  problems. 

2.  Slate  the  status  of  each  goal  scheduled  to  be  achieved  during  the  report  period. 

3.  State  the  corrective  action  planned  to  resolve  implementation  problems  and  state  the 
effect  of  these  problems  on  the  remaining  schedule  for  achieving  remaining  project  goals. 

4.  If  appropriate,  identify  changes  needed  in  the  Implementation  plan  specified  in  Parts  III 
and  IV  of  the  grant  application  to  overcome  problems.  Changes  which  alter  plans  and/or 
goals  set  forth  in  Part  III  or  IV  of  the  application  require  prior  grantor  agency  approval  and 
the  Issuance  of  a  Grant  Adjustment  Notice. 

5.  State  virhat  technical  assistance  the  grantor  agency  might  provide  during  the  coming 
quarter  to  help  resolve  Implementation  problems.  If  technical  assistance  has  been 
provided  to  resolve  implementation  problems,  state  the  problems  (or  tasks)  addressed 
and  the  results  (or  impact)  of  the  assistance  provided. 

6.  Based  on  the  perfomnance  measures  In  the  grant  application  (Implementation  plan), 
indicate  In  quantitative  terms  the  results  (of  the  project)  achieved  both  during  the 
reporting  period  and  cumulative-to-date.  Explanatory  and  qualifying  statements  will  be 
helpful  here,  especially  if  project  objectives  have  changed. 

e.  Special  Requirenfients  .  Special  reporting  requirements  or  instructions  may  be  prescribed  for 
categorical  projects  in  certain  program  or  experimental  areas  to  better  assess  impact  and 
comparative  effectiveness  of  the  overall  categorical  grant  program.  These  will  be 
communicated  to  affected  grantees  by  the  agency. 


Public  reporting  burden  for  this  collection  of  information  Is  estimated  to  average  2  hours  per  response,  Including  the 
time  for  reviewing  Instructions,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and 
completing  and  reviewing  the  collection  of  Information.  Send  comments  regarding  this  burden  estimate  or  any  other 
aspects  of  this  collection  of  information.  Including  suggestions  for  reducing  this  burden,  to  the  U.  S.  Department  of 
Justice,  Office  of  Justice  Programs,  810  Seventh  Street,  N.W.,  Washington,  D.C.  20531;  and  to  the  Public  Use  Reports 
Project,  1 121-0140,  Office  of  Information  and  Regulatory  Affairs,  Office  of  Management  and  Budget,  Washir^gton,  D.C. 
20503.  Do  NOT  send  completed  Progress  Reports  to  OMB. 
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Twice  a  year,  the  Office  of  Juvenile  Justice  and  Delinquency  Prevention 
(OJJDP)  grantees  are  required  to  complete  a  Categorical  Assistance 
Progress  Report — a  narrative  report  that  is  to  include  a  summary  of  the 
status  of  their  particular  projects’  goals,  quantitative  project  results  based 
on  performance  measures  set  forth  in  their  grant  applications,  actions 
planned  to  resolve  any  implementation  problems,  and  any  technical 
assistance  they  might  need.  In  8  of  the  16  msgor  programs  we  reviewed, 
grantees  received  only  this  general  guidance,  and  were  not  subject  to  any 
additional  reporting  requirements.  In  the  other  eight  programs  we 
reviewed,  grantees  were  required  to  foUow  this  standard  guidance  and,  in 
addition,  report  more  specific  information.  Grantees  in  all  16  programs 
reported  input,  output,  and/or  outcome  data  related  to  the  process, 
implementation,  and/or  accomplishments  of  their  projects,  such  as 
acquisition  of  additional  funding  for  a  project  evaluation,  the  number  of 
project  participants,  or  the  number  of  missing  children  recovered. 

Table  7  provides  summary  information  on  the  eight  programs  in  which 
grantees  are  not  subject  to  additional  reporting  requirements  and 
examples  from  grantees’  progress  reports.  Table  8  provides  similar 
information  regarding  the  other  eight  programs  in  which  grantees  are 
additionally  required  to  report  specified  data  to  OJJDP  or  outside 
evaluators,  as  well  as  the  specific  performance  measures  on  which 
grantees  are  required  to  report.  Unless  otherwise  noted,  the  examples  of 
reported  information  represent  individual  grantee  or  subgrantee  data^  for  a 
6-month  period.  Information  provided  regarding  the  specific  data  on  which 
grantees  are  required  to  report  do  not  necessarily  include  aU  performance 
data  required. 


^  Examples  from  OJJDP’s  five  formula/block  grant  programs  were  typically  reported  at  the 
subgrantee  or  community  level. 
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Table  7:  Eight  Programs  GAO  Reviewed  With  General  Reporting  Requirements  Only 


Program 
name _ 

State  Challenge 
Activities 


Community 
Prevention  Grants 
Program 
(Title  V) 


Juvenile 
Accountability 
Incentive  Block 
Grants  Program 


Active 

grantees 

(FY2000)  Program  description _ 

52  This  program,  established  in  1992  under 
Title  II,  Part  E  of  the  Juvenile  Justice  and 
Delinquency  Prevention  Act  of  1974,  as 
amended,  provides  incentive  grants  to 
states  to  improve  their  juvenile  justice 
system.  States  must  use  funds  to  develop, 
adopt,  or  improve  policies  in  1 0  specified 
challenge  activity  areas,  including  (1)  basic 
system  services,  (2)  access  to  counsel,  (3) 
community-based  alternatives,  (4)  violent 
juvenile  offender  facilities,  (5)  gender- 
specific  policies  and  programs,  (7)  state 
ombudsman,  (8)  deinstitutionalization  of 
status  offenders,  (9)  alternatives  to  school 
suspension  and  expulsion,  and  (10) 
aftercare  services/state  agency 
coordination/case  review  system.  To  receive 
these  grants,  states  must  also  participate  in 
the  Title  II,  Part  B  Formula  Grants  Program. 
States  may  subaward  State  Challenge 
Activities  funds  to  communities. 


54  This  program  funds  collaborative, 

community-based  delinquency  prevention 
efforts.  The  program  integrates  six 
fundamental  principles — comprehensive 
and  multidisciplinary  approaches,  research 
foundation  for  planning,  community  control 
and  decision  making,  leveraging  of 
resources  and  systems,  evaluation  to 
monitor  program  effectiveness,  and  a  long¬ 
term  perspective — ^that  combine  to  form  a 
strategic  approach  to  reducing  juvenile 
delinquency. 


54  The  Congress  established  this  formula  grant 
program  in  1997  to  promote  greater 
accountability  among  juveniles  who  are 
involved  in  the  juveniles  justice  system.  To 
qualify  for  funds,  states  must  demonstrate  a 
commitment  to  increasing  such 
accountability.  Funds  are  allocated  to  states 
based  on  their  relative  population  of  youths 
under  age  1 8,  and  states  must  pass  on  at 


Examples  of  information  reported  by  grantee(s) 

per  general  requirements _ _ _ 

.  Implemented  an  aftercare  program  for  state  wards 
returning  to  their  community  that  provided 
community  reintegration  services  including 
counseling,  employment  readiness  services,  family 
mediation,  and  weekly  check-in  and  monitoring 
programs. 

•  Hired  a  disproportionate  minority  confinement 
coordinator. 

•  Home  visitation  research  contract  delayed  due  to 
paperwork  complications. 

•  Held  weekly  meetings  with  female  juveniles 
identified  by  the  court  to  receive  counseling  and 
support  through  a  gender-specific  services 
program. 

•  Selected  and  made  subaward  to  private  provider 
with  experience  in  developing  intake  services  for 
youths  cited  by  law  enforcement  with  a  focus  on 
preventing  unnecessary  placement  in  attendant 
care  or  detention. 

•  Provided  quarterly  training  to  all  case  management 
staff  on  more  rapidly  and  successfully  returning 
youths  in  state  custody  from  out-of-home 
placement  into  community  and  in-home  placement. 

•  Exceeded  goal  of  providing  career  exploration  and 
training  on  work  awareness  skills  to  150  high 
school  students. 

•  Trained  1 1  students  to  become  mentors. 

•  Provided  substance  abuse  education  to  six  middle 
school  girls. 

•  Experienced  difficulties  starting  a  delinquency 
prevention  program  to  develop  self-esteem  and 
reduce  inappropriate  school  behavior  in  at-risk 
seventh-grade  students. 

•  Hired  two  outreach  workers  to  work  with  elementary 
and  junior  high  schools. 

•  Created  an  instructional  video  on  quality  daycare 
issues  and  distributed  it  to  parents  and  the 
community. 

•  Enrolled  and  “graduated”  13  children  in  a  6-week 

kindergarten  preparation  class. _ 

•  Hired  a  case  manager  to  monitor  attendance  and 
grades  of  juveniles  in  a  youthful  offender  program. 

•  A  local  office  declined  to  participate  in  the 
mentoring  program  and  the  subgrantee  is 
considering  alternative  participants. 

•  Completed  a  study  of  achieving  a  system  of 
graduated  sanctions  within  the  state. 

•  Enrolled  30  youths  in  a  juvenile  drug  testing 
program  and  performed  52  drug  screens. 
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Program 

name 


Enforcing  the 
Underage 
Drinking  Laws 
Program  (block 
grant  component) 


Enforcing  the 
Underage 
Drinking  Laws 
Program 
(discretionary 
grant  component)® 


Active 

grantees 

(FY2000)  Program  description _ 

least  75  percent  of  their  funds  to  local 
governments,  absent  a  waiver.  Funds  can 
be  used  for  12  program  purpose  areas. 


51  The  block  grant  component  of  the  Enforcing 
the  Underage  Drinking  Laws  Program  helps 
states  and  the  District  of  Columbia  develop 
comprehensive  and  coordinated  efforts  to 
enforce  state  laws  that  prohibit  the  sale  of 
alcoholic  beverages  to  minors  and  to 
prevent  the  purchase  or  consumption  of 
alcoholic  beverages  to  those  under  age  21 . 
Grantees  may  use  funds  to  support 
activities  In  one  or  more  of  three  priority 
areas,  including  (1)  law  enforcement  and 
prosecution  efforts  that  target 
establishments  suspected  of  selling  alcohol 
to  minors;  (2)  public  advertising  campaigns 
to  educate  businesses  and  youths  about  the 
prohibitions  against  illegal  alcohol  sales  and 
purchases;  and  (3)  other  innovative 
programs  to  prevent  and  combat  underage 
drinking. 


23  The  discretionary  grant  component  of  the 
Enforcing  the  Underage  Drinking  Laws 
Program  provides  grants  to  states  and 
territories  to  support  the  block  grant 
component  of  the  same  program.  Its 
purpose  is  to  expand  the  number  of 
communities  taking  a  comprehensive 
approach  to  the  problem  of  underage 
drinking,  with  an  emphasis  on  increasing 
law  enforcement  activity  regarding  the  sale 
of  alcohol  to  minors.  States  that  receive 


Examples  of  information  reported  by  grantee(s) 

per  general  requirements _ 

•  Expanded  a  drug  abuse  program  for  juveniles 
referred  to  the  court  to  include  a  second  group 
involving  both  parents  and  juveniles.  To  date,  the 
project  has  held  3  groups  of  6-week  classes. 

•  No  juveniles  completed  a  6-week  intensive 
educational  program  for  substance  abuse  during 
the  reporting  period  because  of  problems  starting 
the  program. 

•  The  placement  of  a  full-time  police  officer  at  a  high 
school  was  delayed  due  to  problems  involving  the 
officer’s  training. 

•  Contracted  with  a  part-time  prosecutor  to  assist  full¬ 
time  prosecutor  with  juvenile  pre-trial  and  trial 
services,  Including  compiling  case  information, 
preparing  witnesses,  and  tracking  cases.  The  part- 
time  prosecutor  works  an  average  of  25  cases  per 
week.  In  total,  about  234  cases  have  been  handled. 

•  Experienced  difficulties  in  recruiting  a  data  analyst 

due  to  problems  with  applicants  meeting  the 
minimum  requirements. _ _ 

•  Formed  1 5  youth  task  groups  in  order  to  examine 
the  community  norms  and  messages  that  young 
people  get  regarding  alcohol. 

•  Placed  billboards  to  educate  the  public  about 
alcohol  use  in  20  communities  for  4  weeks. 

•  Produced  a  60-second  radio  spot  aimed  at 
preventing  underage  alcohol  use. 

•  T rained  20  community  teams  about  prevention 
approaches  to  reduce  underage  drinking. 

•  Established  a  support  position  to  improve 
coordination  and  communication  between 
enforcement,  prevention,  and  treatment  systems. 

•  Held  a  full-day  conference  for  established  task 
force  groups. 

•  Experienced  implementation  problems  due  to 
problems  with  students’  schedules  and  limitations 
of  the  school  year  schedule. 

•  The  local  public  television  station  worked  with  a 

youth  task  force  group  to  design  and  film  a  1-hour 
documentary  about  underage  drinking. _ 

•  Three  communities  developed  a  compliance  check 
database  form  to  be  used  by  law  enforcement 
agencies. 

•  Requested  technical  assistance  for  a  1  -day  training 
session  for  local  coalitions  on  environmentally- 
based  activities. 

•  Published  two  reports  on  status  of  underage 
drinking  in  targeted  community. 

•  Held  two  advisory  committee  meetings. 

•  Implemented  new  diversion  program  for  underage 
alcohol  violators  and  achieved  a  32-percent 
success  rate  for  youths  completingjhe_pr^^ 
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Program 

name 


Active 

grantees 

(FY2000) 


Tribal  Youth 
Program 


Model  Courts 
Program^ 


Program  description _ 

these  competitive  awards  select  between 
two  and  eight  diverse  jurisdictions  as 
subgrantees. 


Under  this  discretionary  award  program, 
federally  recognized  Tribes  can  apply  for 
funds  to  (1)  reduce,  control,  and  prevent 
crime  both  by  and  against  tribal  youths;  (2) 
provide  interventions  for  court-involved  tribal 
youths;  (3)  improve  tribal  juvenile  justice 
systems;  and  (4)  provide  prevention 
programs  focusing  on  alcohol  and  drugs. 
The  program  is  intended  to  provide  a 
flexible  funding  stream  for  tribes. 


This  program  awards  one  earmarked®  grant 
annually  to  the  National  Council  of  Juvenile 
and  Family  Court  Judges  to  provide  training 
and  technical  assistance  to  23  “model 
courts”  around  the  nation  that  process  child 
abuse  and  neglect  cases.  The  program’s 
oal  is  to  design  and  deliver  technical 


Examples  of  information  reported  by  grantee(s) 

per  genera!  requirements _ _ 

•  Experienced  difficulty  In  bringing  one  subgrantee  on 
board;  the  subgrantee  has  not  yet  attended  any 
planning  sessions. 

•  Four  communities  hired  part-time  program 
coordinators. 

•  Conducted  six  monthly  conference  calls  with  four 
communities. 

•  Youth  coalition  placed  4,000  warning  stickers  on 
alcoholic  beverages  in  13  stores. 

•  Youth  coalition  designed  placemats  containing 
underage  drinking  facts  and  figures  for  restaurant 
and  grocery  store  distribution. 

•  Published  two  newsletters. 

•  Was  not  able  to  hold  banquet  recognizing  top  1 00 
businesses  that  had  discouraged  underage  alcohol 
sales  due  to  preparation  demands  of  an  anti¬ 
alcohol  youth  conference. 

•  Arranged  for  more  than  700  youths  to  sign  a  pledge 

not  to  drink. _ _ _ 

•  Began  keeping  juvenile  crime  data  for  future 
analysis  of  whether  targeted  reductions  have  been 
achieved. 

•  Drafted  court  diversion  protocol. 

•  Identified  tribal  judges  and  court  clerks. 

•  Experienced  set-backs  in  establishing  four 
functional  tribal  courts  to  deal  with  juvenile  crime  for 
reasons  including  the  resignation  of  the  general 
counsel  and  competing  demands  of  the  commercial 
and  subsistence  fishing  season. 

•  Reported  conferences  and  training  events 
attended. 

•  Met  with  state  representatives  about  possible 
transfers  of  cases  from  state  to  tribal  court. 

•  Began  identifying  courtroom  equipment  needs  and 
researching  purchases. 

•  Obtained  examples  of  tribal  juvenile  codes. 

•  Taught  conflict  resolution  skills  to  families  upon 
request. 

•  Suffered  a  set-back  in  reducing  cases  of 
misbehavior  and  rules  violations  in  the  schools 
because  the  community  truancy  officer  resigned 
and  had  not  yet  been  replaced. 

•  Referred  five  youths  for  counseling  as  a  result  of 
conflict  resolution  curriculum  presented  in  schools. 

Information  reported  by  grantee  through  progress 

reports: 

•  Made  31  on-site  staff  visits  to  model  courts. 

•  Held  96  national,  regional,  state,  and  local  training 
events  for  nonmodel  court  jurisdictions. 

•  Supported  the  establishment  of  two  model  courts. 

•  Trained  9,729  model  court  and  nonmodel  court 
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Program 


Active 

grantees 

(FY2000) 


National 
Clearinghouse 
and  Resource 
Center  for  Missing 
and  Exploited 
Children® 


Program  description _ 

assistance  and  training  programs  that 
improve  dependency  courts’  handling  of 
such  cases.  Model  courts  serve  as  national 
“laboratories”  for  implementing  systems- 
wide  change  in  the  processing  of  abuse  and 
neglect  cases.  Reforms  and  Initiatives  vary 
across  courts.  The  Council  also  supports  a 
mentoring  program  in  which  experienced 
model  court  personnel  “mentor”  and  train 
personnel  from  new  model  courts. 


1  The  National  Center  for  Missing  and 
Exploited  Children  (NCMEC)  Is  the  national 
resource  center  and  clearinghouse 
dedicated  to  missing  and  exploited 
children’s  issues.  By  law,  NCMEC  is  to  (1) 
establish  a  toll-free  hotline,  (2)  provide 
information  on  free  or  low-cost 
transportation  for  missing  children  and  their 
families,  (3)  coordinate  programs  that 
reunite  missing  children  with  their  families, 
(4)  disseminate  information  that  benefits 
missing  and  exploited  children,  and  (5) 
provide  training  and  technical  assistance  to 
law  enforcement  and  other  agencies. 


Examples  of  information  reported  by  grantee(s) 

per  general  requirements _ _ 

personnel. 

•  Distributed  1 7,81 8  technical  assistance  bulletins. 

•  Conducted  1 34  telephone  consultations. 

•  Distributed  2,341  resource  packets. 

Information  reported  by  grantee  through  annual  status 

report:" 

•  Reported  one  court  reduced  its  caseload  of  58,000 
abused  and  neglected  children  in  1 995  (prior  to 
program  implementation)  to  27,001  children  In 
1999. 

•  Reported  one  court  reduced  the  average  length  of 
abuse  and  neglect  complaint  cases  from  3.5  years 
in  court  year  1998  to  2.4  years  in  court  year  1999. 

•  Reported  one  court  established  a  child  protection 
division  with  two  judges  who  hear  all  child 
protection  cases. 

•  Following  this  change,  the  average  case  length 

(from  petition  to  adjudication)  for  one  section  of  the 
division  dropped  from  140  days  to  between  29  and 
41  days. _ ^ _ _ 

•  Upgraded  toll-free  hotline. 

•  Received  24,983  hotline  calls. 

•  Assisted  in  recovering  1 ,61 0  missing  children. 

•  Described  missing  children  cases  in  which  NCMEC 
assisted. 

•  Distributed  5,306  pictures  of  missing  children. 

•  Described  public  awareness  campaigns. 

•  Trained  1 ,708  law  enforcement  and  health  care 
personnel. 

•  Described  joint  projects  with  other  agencies. 

•  Completed  48  age  progressions  of  photographs. 

•  Received  5,291  child  pornography  tips  online,  and 
492  tips  through  the  hotline. 

•  Described  technical  assistance  provided  to  other 
aaencies. 


“This  program  was  established  In  fiscal  year  1 998  under  the  name  Combating  Underage  Drinking. 

^OJJDP  has  emphasized  to  the  single  grantee  for  this  program  that  it  should  include  quantitative 
performance  data  in  Its  progress  reports,  but  did  not  prescribe  the  specific  indicators  on  which  the 
grantee  must  report. 

"Earmarked  grants  are  those  for  which  the  Congress  sets  aside  a  specified  amount  for  an  identifiable 
grantee  and/or  a  specific  authorized  purpose. 

‘The  grantee  chooses  to  provide  performance  data  regarding  model  court  activities  and 
accomplishments  in  an  annual  status  report  that  is  available  to  nonmodel  courts  and  to  the  general 
public. 

“Although  the  grantee  in  this  program  is  only  required  to  submit  progress  reports  semiannually,  it 
chooses  to  submit  progress  reports  on  a  quarterly  basis.  Examples  presented  in  this  table  were 
excerpted  from  a  recent  quarterly  progress  report. 

Source:  GAO  analysis  of  OJJDP  and  grantee  data. 


Page  43 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Appendix  IV:  Descriptions  of  OJJDP 
Programs,  Reporting  Requirements,  and 
Examples  of  Reported  Information 


Table  8:  Eight  Programs  GAO  Reviewed  With  Specific  Reporting  Requirements  and  General  Reporting  Requirements 


Program 
name _ 

Formula  Grants 
Program 


Active 

grantees 

(FY2000)  Program  description _ 

54  OJJDP  has  administered  this 
program  since  1975.  The 
program  provides  funds  to 
help  states,  U.S.  territories, 
and  the  District  of  Columbia 
implement  a  comprehensive 
state  juvenile  justice  plan 
based  on  a  detailed  study  of 
needs  to  support  state  and 
local  delinquency  prevention 
and  intervention  efforts  and 
juvenile  justice  system 
improvements.  To  be  eligible 
for  a  formula  grant,  a  state 
must  designate  an  agency  to 
prepare  a  3-year  plan, 
establish  a  State  Advisory 
Group,  and  commit  to 
complying  with  the  four  core 
requirements  of  the  Juvenile 
Justice  and  Delinquency 
Prevention  Act  of  1974,  as 
amended. 


Comprehensive 
Gang  Initiative 


2  This  program  assists 
community  coalitions  in 
implementing  and  testing  the 
comprehensive  gang  model— 
a  collection  of  strategies 
aimed  at  the  prevention, 
Intervention,  and  suppression 
of  youth  gang  crime  and 
violence.  The  model  requires 
communities  to  assess  local 
gang  problems,  identify  key 
risk  factors  among  youths, 
develop  program  plans,  and 


Specific  data  required  from 
and  reported  by  program 
grantee(s)  _ 

Grantees  (states)  must  report 
data  including  but  not  limited 
to  the  following: 

•  Number  of  accused  status 
offenders  and  nonoffenders 
held  in  any  secure 
detention  or  correctional 
facility  for  more  than  24 
hours. 

•  Number  of  accused  status 
offenders  and  nonoffenders 
securely  detained  in  any 
adult  jail,  lockup,  or 
nonapproved  collocated 
facility  for  any  period  of 
time. 

•  Number  of  accused  status 
offenders  and  nonoffenders 
placed  in  facilities  not  near 
their  home,  not  the  least 
restrictive  appropriate 
alternative,  and  not 
community-based. 

•  Number  of  juvenile 
offenders  and  nonoffenders 
not  separated  from  adult 
criminal  offenders  in 
facilities  used  for  secure 
detention  and  confinement 
of  both  juveniles  and  adults. 

•  Number  of  juveniles 
detained  in  collocated 
facilities  not  approved  by 
the  state  that  were  not  sight 
and  sound  separated  from 
adult  criminal  offenders. 

•  Number  of  juveniles  served. 

•  Age,  sex,  and  income  level 
of  participating  youths. 

•  School  participation  and 
academic  performance  for 
participating  youths. 

•  Criminal  activity,  gang 
activity,  drug  use,  and 
alcohol  use  by  participating 
youths. 

•  Project  staff’s  perceptions 
of  the  seriousness  of  gang 
and  nongang  problems  in 


Examples  of  information  reported 

by  grantee(s)  per  general 

requirements _ 

•  For  21  youths  involved  in  a 
truancy  abatement  program, 
school  attendance  increased  by  an 
average  of  89  percent. 

•  A  runaway  youth  program 
provided  counseling  to  129 
runaway  youths  and  their  families; 
of  57  cases  closed,  86  percent  of 
youths  returned  to  home,  a  friend, 
or  a  relative. 

•  Discontinued  a  violence 
prevention  curriculum  for  middle 
and  high  school  students  and  a 
conflict  resolution  training  program 
for  eighth-grade  students  due  to 
the  programs’  lack  of 
effectiveness. 

•  Twenty-six  status  offenders  and 
their  families  participated  in  family 
counseling  activities  that  focused 
on  parenting  skills, 
communication,  and  prosocial 
behaviors. 

•  A  mentoring  program  has 
continued  to  have  difficulty  in 
recruiting  appropriate  mentors. 

•  Enrolled  40  youths  pending 
adjudication  and/or  disposition  in 
home  detention  programs  in  lieu  of 
Incarceration,  shelter,  or  detention 
placement. 

•  Despite  efforts,  was  unable  to 

gather  and  analyze  data  on  youths 
participating  in  a  tutoring  program 
due  to  a  lack  of  school  system 
cooperation. _ 

•  Received  university  grant  to  fund 
local  project  evaluation. 

•  Developed  action  team  to  address 
problem  of  female  gang  members. 

•  Project  youths  were  not 
participating  in  tattoo  removal 
program;  project  staff  need  to 
better  promote  benefits  of  tattoo 
removal. 

•  Recruited  five  new  community 
leaders  to  join  project. 

•  Held  biweekly  staff  meetings  to 
discuss  progress  of  participating 


Page  44 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Appendix  IV:  Descriptions  of  OJJDP 
Programs,  Reporting  Requirements,  and 
Examples  of  Reported  Information 


Program 

name 


Rural  Gang 
Initiative 


Juvenile 

Mentoring 

Program 

(JUMP) 


Active 

grantees 

(FY2000)  Program  description _ 

evaluate  program  outcomes. 
Coalitions  focus  on  five  core 
strategies:  community 
mobilization,  academic  and 
economic  opportunities,  social 
intervention,  gang 
suppression,  and 
organizational  change  and 
development. 


Specific  data  required  from 
and  reported  by  program 

grantee(s)  _ 

the  community. 

•  Nature  and  frequency  of 
services  provided  to  youths 
and  families. 

•  Youth  workers’  (probation 
officers,  case  workers) 
perceptions  of  the 
effectiveness  of  services 
provided  to  youths  and 
families. 


4  A  spin-off  of  the 

Comprehensive  Gang  Initiative 
that  adapts  the  comprehensive 
gang  model  to  rural 
communities.  Two  sites  have 
incorporated  the  full  version  of 
the  model  and  two  sites  follow 
a  modified  version  of  the 
Comprehensive  Gang 
Initiative. 


Aggregated  demographic 
data  for  community 
(unemployment  rates, 
education  attainment,  race). 
Community  attitude  data 
(community’s  perception  of 
gang  problem). 

Youth  attitude  data 
(perceptions  about  gang- 
involved  youths  from  at-risk 
youths). 

Law  enforcement  crime 
data  (data  on  nature  of 
criminal  gang  activity  in 
community— number  of 
assaults,  property  crimes, 
etc.) 

Community  resource  data 
(inventory  of  services 
available  to  gang  and 
community  members — e.g., 
conflict  resolution,  individual 
counseling,  etc.) 


175  JUMP  provides  one-to-one 
mentoring  for  youths  at  risk  of 
delinquency,  gang 
involvement,  educational 
failure,  or  dropping  out  of 
school.  Awards  are  made  to 
local  educational  agencies  or 
private  or  nonprofit  agencies 
that  have  clearly  defined 
relationships  with  local 
schools.  Grantees  must 
conduct  thorough  background 
checks  on  volunteer  mentors 
to  ensure  a  safe  environment 


Youth  information: 

•  Demographic  information, 
such  as  age,  gender,  and 
ethnicity. 

•  Living  situation. 

•  Medical  or  mental  health 
problems. 

•  Total  number  of  school 
absences. 

Mentor  information: 

•  Demographic  Information 
such  as  age,  gender,  and 
ethnicity. 

•  Reason  for  becoming 


Examples  of  information  reported 

by  grantee(s)  per  generai 

requirements _ 

youths. 

•  Inadequate  staffing  levels  caused 
delays  in  processing  higher  than 
expected  numbers  of  youth 
referrals  and  prevented  project 
expansion  to  additional 
neighborhoods. 

•  Revised  staff  procedures  for 
delivering  and  coordinating 
warrants. 

•  Recruited  25  girls  for  selected 

projects. _ 

•  Experienced  delay  in  hiring  a 
project  coordinator  and  outreach 
staff  due  to  a  lack  of  qualified 
applicants;  project  is  preparing  for 
second  recruitment  effort. 

•  Developed  steering  committee 
composed  of  representatives  from 
community  organizations  involved 
with  youths. 

•  Attended  training  on 
implementation  of  the 
comprehensive  gang  model. 

•  Located  and  rented  a  building  to 
house  project  staff. 

•  Insufficient  staff  to  follow  up  on 
court  orders  against  targeted 
youths  who  were  at-risk  for  joining 
a  gang. 

•  In  the  process  of  forming  an 
intervention  team  composed  of  law 
enforcement,  school,  and  social 
services  personnel. 

•  Hired  a  gang  crime  specialist. 

•  Computer  failure  prevented  police 

department  from  providing  crime 
data. _ _ 

•  Exceeded  goal  of  matching  50  at- 
risk  girls  with  mentors. 

•  Mentors  and  mentees  participated 
In  classes  highlighting  other 
cultures. 

•  Retained  50  percent  of  mentors 
from  the  previous  year. 

•  Experienced  problems  with 
parents  who  were  unwilling  to 
complete  project  requirements, 
such  as  interviews  and  home 
visits. 

•  Continued  to  recruit  mentors  from 
local  businesses. 
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Program 

name 


SafeFutures 


Drug-Free 

Communities 

Support 

Program 


Active 

grantees 

(FY2000)  Program  description _ 

for  each  child.  JUMP  also 
emphasizes  projects  that  are 
established  in  communities 
where  60  percent  or  more  of 
the  youths  qualify  to  receive  a 
free  or  reduced-price  lunch. 


6  This  program,  based  on 
OJJDP’s  comprehensive 
strategy  model,  targets 
communities  that  have  already 
begun  to  reform  their  juvenile 
care  systems.  Program  goals 
include  developing  a  full  range 
of  graduated  sanctions, 
ensuring  community  safety, 
and  providing  treatment 
services  for  juveniles. 
Communities  use  one  grant  to 
tap  into  nine  OJJDP  program 
components  (e.g.,  juvenile 
mentoring,  aftercare,  mental 
health,  delinquency 
prevention,  gang  programs, 
etc.) 


307  This  program  is  intended  to 
increase  citizen  participation 
and  strengthen  community 
anti-drug  coalition  efforts  to 
reduce  substance  abuse 
among  youths  in  communities 
throughout  the  United  States 
and,  over  time,  to  reduce 
substance  abuse  among 
adults. 


Specific  data  required  from 

and  reported  by  program 

grantee(s)  _ 

involved  in  the  mentoring 
program, 

•  Highest  education 
completed. 

•  Previous  mentoring 
experience. 

•  Demographic  data  (age, 
race,  sex)  for  youth 
participants. 

•  Family/household 
characteristics 
(employment,  number  of 
family  members,  number  of 
juveniles/adults  living  at 
home,  school  history,  family 
rules)  for  youth  participants. 

•  Self-reported  data  by  youth 
participants  (neighborhood 
characteristics,  gang 
affiliation,  number  of 
delinquent  acts, 
characterization  of  family 
relationships,  drug  and 
alcohol  abuse,  emotional 
problems,  participation  in 
project  activities,  drug  and 
alcohol  abuse  by  family 
members). 


•  Number  of  organizations 
represented  on  the 
grantee’s  coalition. 

•  Types  of  organizations 
represented  on  the 
coalition. 

•  Number  of  individuals  who 
are  coalition  members. 

•  Number  of  youths,  parents, 
and  community  leaders  who 
are  coalition  members. 

•  Number  of  recruitment 
activities  held. 

•  Steps  taken  to  engage 
members  in  drug  prevention 
planning. 

•  Activities  sponsored  by  the 
coalition. 

•  Number  of  participants  in 

activities^ _ 


Examples  of  information  reported 

by  grantee(s)  per  general 

requirements _ _ 

•  Twenty  out  of  26  mentors  who 
responded  to  a  survey  said  the 
program  met  their  expectations 
and  23  said  they  would  participate 
again. 

•  Twenty-five  youths  dropped  out  of 

a  project.  _ 

•  Developed  and  submitted 
applications  for  additional  funding. 

•  Experienced  problems  providing 
substance  abuse  treatment 
services  due  to  the  lack  of  a  full¬ 
time,  trained  substance  abuse 
counselor. 

•  Hired  a  new  local  evaluator. 

•  Referred  249  juveniles  for  direct 
services  and  care  coordination. 

•  Completed  1 3  community  service 
projects  involving  394  students 
and  families. 

•  Experienced  problems 
implementing  a  mentoring 
program — i.e.,  recruiting  mentors, 
retaining  matches,  and  motivating 
mentors  to  attend  sessions. 

•  Continued  coordination  between 
law  enforcement  agencies  on 
gang  intelligence  resulted  in  the 
arrest  of  two  arson  suspects. 

•  Installed  a  computer  system  to 
track  gang  members  and  gang- 
related  activities  in  the  community. 

•  Exceeded  goal  of  1 00  community 
organizations  participating  in 
workshops. 

•  Produced  anti-smoking 
commercial. 

•  Invited  health  professionals  to  a 
strategic  planning  meeting. 

•  Staff  worked  with  drug-free  clubs. 

•  Hired  first  full-time  staff. 

•  Printed  quarterly  newspaper. 

•  Distributed  20,000  copies  of  a 
family  resource  directory. 

•  Experienced  low  attendance  at  a 
strategic  planning  meeting. 

•  Provided  drug  prevention 
education  to  78  after-school 
detention  students. 
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Program 

name 


Internet  Crimes 
Against 
Children  Task 
Force  Program 


Children’s 

Advocacy 

Centers 


Active 

grantees 

(FY2000)  Program  description 


50  This  program  encourages 
communities  to  develop 
regional,  multi-jurisdictional, 
and  multi-agency  responses  to 
online  enticement  and  child 
pornography  cases.  These 
task  forces  provide  forensic, 
prevention  and  Investigative 
assistance  to  parents,  law 
enforcement,  prosecutors, 
educators,  and  other 
professionals  working  on  child 
victimization  Issues. 


This  program  provides  training 
and  technical  assistance  to 
local  children’s  advocacy 
centers  and  promotes  their 
expansion  nationwide. 
Children’s  advocacy  centers 
coordinate  law  enforcement 
and  protective  services 
responses  to  victims  of  child 
abuse  through  multi¬ 
disciplinary  teams.  Under  this 
program  four  regional  centers 
and  one  national  association 
provide  training  and  technical 
assistance  to  local  centers;  the 
national  association  also 
awards  subgrants  to  local 
centers  for  training,  program 
development,  and  other 
purposes. 


Specific  data  required  from 
and  reported  by  program 
grantee(s)  _ 

•  Target  population  for 

activities. _ 

Grantees  in  this  program  must 
complete  a  monthly 
performance  measures  report 
that  collects  information 
including,  but  not  limited  to: 

•  Number  of  arrests. 

•  Number  of  child 
pornography  producer 
cases. 

•  Number  of  child 
pornography  seller  cases. 

•  Number  of  search  warrants. 

•  Number  of  subpoenas. 

•  Number  of  referrals  from 
other  agencies. 

•  Number  of  law  enforcement 
personnel  trained. 

•  Number  of  presentations 

given  to  students,  parents, 
teachers,  citizens  or  the 
media. _ 

Training  and  technical 
assistance  grantees  are 
required  to  report  on: 

•  Number  of  training  sessions 
conducted. 

•  Number  of  practitioners 
trained. 

•  Number  of  technical 
assistance  requests  filled. 

•  Number  of  consultations 
(on-site,  electronic, 
telephone)  provided. 

•  Number  of  publications 
developed  and  distributed. 

National  grantee  is  required  to 
report  subgrantee  data  on: 

•  Number  of  new  and 
developing  local  centers. 

•  Number  of  on-site 
monitoring  visits. 

•  Number  of  grant 
applications  approved. 

•  Type  of  grants  approved 
(program  development, 
program  support,  training, 
and  state  chapter). 


Examples  of  information  reported 
by  grantee(s)  per  general 
requirements  _ 


Provided  summaries  of  24  cases 
during  the  reporting  period. 

Hired  a  child  victim  service 
specialist. 

Achieved  goal  of  implementing 
training  seminars  for  local,  states 
and  federal  law  enforcement 
officers. 

Acquired  necessary  equipment  for 
investigating  online  offenders  such 
as  desktop  and  laptop  computers, 
cameras,  and  investigative 
software. 

Made  presentations  on  Internet 
safety  to  teachers,  parents, 
community  groups,  and  school- 
age  children. 


Training  and  technical  assistance 

grantees: 

•  Finalized  a  diversity  plan  to  ensure 
that  children  receive  services  in  a 
culturally  appropriate  setting. 

•  Experienced  difficulties  agreeing 
on  research  methods  to  evaluate 
the  children’s  advocacy  centers’ 
impact  on  case  outcomes. 

•  Developed  a  “multidisciplinary 
team  development’  curriculum. 

•  Identified  and  provided  assistance 
to  underserved  rural  communities. 

National  grantee: 

•  Reviewed  subgrantees’  biannual 
reports  on  activities  held  during 
the  reporting  period. 

•  Unexpected  resignation  of  grants 
coordinator  hampered  data 
collection  efforts  and  the  awarding 
and  processing  of  grants. 

•  Approved  409  applications  for 
subgrants  and  notified  recipients. 

•  Surveyed  local  accredited  centers 
and  found  that  49,350  juveniles 
had  been  served  during  the 
reDortinq  period. 
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The  Office  of  Juvenile  Justice  and  Delinquency  Prevention’s  (OJJDP) 
training  and  technical  assistance  programs  and  research  programs  are 
unique  in  that  they  cut  across  many  of  OJJDP’s  other  programs.  Also, 
grantees^  in  each  of  these  two  areas  typically  report  the  same  types  of 
quantitative  performance  data  as  other  grantees  in  their  area,  even  though 
OJJDP  does  not  usually  prescribe  the  specific  performance  measures  on 
which  the  grantees  should  report.  Training  and  technical  assistance 
grantees  maintain  the  same  types  of  data  due  to  the  common  support 
services  they  provide,  and  research  grantees  do  the  same  because  they 
share  a  common  goal  of  producing  research  products. 


Performance  Data  for 
Training  and 
Technical  Assistance 
Grantees 


OJJDP  awards  grants  to  training  and  technical  assistance  providers  to 
support  grantees  in  many  of  OJJDP’s  grant  programs.  OJJDP  administers 
the  vast  majority  of  its  training  and  technical  assistance  grants  through 
three  of  the  Office’s  divisions:  (1)  the  Training  and  Technical  Assistance 
Division  (TTAD),  (2)  the  State  and  Tribal  Assistance  Division  (STAD),  and 
(3)  the  Cluld  Protection  Division  (CPD).^  Many  of  the  training  and 
technical  assistance  providers  are  required  to  report  information  on  their 
projects’  activities  and  accomplishments  semiannually  using  OJP’s 
Categorical  Assistance  Progress  Report  form,  as  do  aU  OJJDP  grantees.^ 
OJP  provides  standard  guidance  on  information  to  be  reported,  such  as 
information  on  the  status  of  each  of  the  grantees’  project  goals  and 
quantitative  results  of  their  projects.  STAD  has  not  imposed  additional  or 
more  specific  reporting  requirements  on  its  training  and  technical 
assistance  providers  and,  for  the  most  part,  neither  have  TTAD"^  nor  CPD.® 


^  OJJDP  awards  grants,  cooperative  agreements,  and  contracts  to  training  and  technical 
assistance  providers,  and  award  grants  and  cooperative  agreements  to  research  providers. 
Officials  commonly  refer  to  all  of  these  as  “grants”  and  to  all  award  recipients  as 
“grantees.” 

^  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  noted  that  these 
divisions  do  not  always  provide  the  same  types  of  services. 

^  AU  training  and  technical  assistance  providers  that  receive  grants  are  required  to 
complete  progress  reports.  Providers  that  are  under  contract  or  have  cooperative 
agreements  with  OJJDP  must  also  report  on  their  accomplishments,  but  not  necessarily 
through  a  progress  report.  Often,  they  are  required  to  report  their  activities  on  a  monthly 
basis. 

^  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  noted  that  TTAD 
requires  Juvenile  Accountability  Incentive  Block  Grants  Program  grantees  to  submit 
monthly  reports.  These  grantees  comprise  approximately  one-fifth  of  all  TTAD  grantees 
and  account  for  approximately  $5  million  annually. 


Page  48 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Appendix  V:  Training  and  Technical 
Assistance  and  Research  Performance  Data 
Reported  by  OJJDP  Grantees 


Officials  explained  that  OJJDP  does  not  require  all  grantees  to  routinely 
report  prescribed  data  because  it  is  reluctant  to  place  additional  reporting 
requirements  on  grantees  due  to  the  Paperwork  Reduction  Act  of  1995,^ 
which  set  goals  to  reduce  the  federal  government’s  reporting  and 
paperwork  burden.  Although  most  of  these  providers  are  not  subject  to 
additional  reporting  requirements  for  prescribed  data,  it  is  not  unusual  for 
them  to  report  on  the  same  or  similar  quantitative  performance  measures. 
Because  of  the  nature  of  the  services  they  provide,  training  and  technical 
assistance  providers  tend  to  maintain  like  data  that  can  readily  be 
counted,  such  as  numbers  of  training  events  held,  practitioners  who 
attended  those  events  (“practitioners  trained”),  and  technical  assistance 
requests  filled.  Some  of  these  providers  also  produce  publications  or 
materials,  such  as  bulletins,  surveys,  curricula,  brochures,  and  other 
support  materials,  and  report  such  information  to  OJJDP.  Table  9 
summarizes  performance  data  we  obtained  regarding  training  and 
technical  assistance  grants.*^ 


®  CPD  has  prescribed  additional,  specific  reporting  requirements  for  providers  in  two  of  its 
programs — ^the  Children’s  Advocacy  Centers  and  the  Missing  and  Exploited  Children 
Training  and  Technical  Assistance  program — such  as  reporting  the  number  of  training 
events  held. 

®  The  Paperwork  Reduction  Act  applies  to  identical  reporting  requirements  imposed  on  10 
or  more  parties.  CPD’s  training  and  technical  assistance  providers  that  have  specific 
reporting  requirements  are  not  affected  by  the  Act  since  they  involve  less  than  10 
providers. 

^  We  attempted  to  obtain  fiscal  year  2000  data  from  aU  grantees;  however,  in  some  cases 
OJJDP  provided  calendar  year  2000  or  fiscal  year  2001  data  due  to  a  lack  of  available  or 
representative  data.  Because  we  obtained  publications  data  that  pertained  only  to  some  of 
the  grantees,  we  did  not  include  such  information  in  the  table.  For  example,  State 
Challenge  Activities  training  and  technical  assistance  providers  reported  producing  10 
bulletins  in  fiscal  year  2001and  Juvenile  Accountability  Incentive  Block  Grant  providers 
reported  that  in  fiscal  year  2000  they  developed  or  were  developing  628  supportive 
materials,  such  as  reports,  curricula,  and  newsletters. 


Page  49 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Appendix  V;  Training  and  Technical 
Assistance  and  Research  Performance  Data 
Reported  by  OJJDP  Grantees 


Table  9:  Training  and  Technical  Assistance  Performance  Data  Reported  by  OJJDP 
Grantees 


Division/Droaram  suDDorted  Traininq  events 

Practitioners 

trained 

Technical 

assistance 

requests 

filled® 

Traininq  and  Technicai  Assistance  Division  (FY2000) _ 

Juvenile  Accountability  Incentive 

Block  Grants  Program 

227 

13,816 

1,374 

Life  Skills  Training^* 

64 

858 

16 

National  Training  and  Technical 

Assistance  Center^ 

d 

d 

329 

Other  programs® 

f 

36,000 

1,633’ 

state  and  Tribai  Assistance  Division  (FY2001)'’ _ _ _ 

Community  Prevention  Grants 

Program  (Title  V) 

55 

2,171 

4 

Formula  Grants  Program 

10 

1,148 

265 

State  Challenge  Activities 

2 

69 

71 

Comprehensive  Strategy  for 

Serious,  Violent,  and  Chronic 

Juvenile  Offenders 

76 

2,252 

232 

Child  Protection  Division  (Calendar  Year  2000) _ _ _ 

Children’s  Advocacy  Centers 

183 

10,939 

6,50r 

Court  Appointed  Special  Advocate 

Program 

194 

5,312 

21,323 

Model  Courts  Program 

161 

42,537 

401 

Permanency  Planning 

124 

38,888 

4,264 

Prosecutor  Training 

f 

9,300 

1,800 

Parents  Anonymous 

3 

149 

f 

Missing  and  Exploited  Children 

Training  and  Technical  Assistance 

68 

2,950 

f 

“Data  can  include  the  number  of  in-depth,  on-site  visits  and/or  the  number  of  responses  to  telephone 
and  e-mail  requests. 


‘’Data  for  this  program  were  from  the  academic  year  2000-2001 ,  which  roughly  corresponds  with  fiscal 
year  2001 . 

The  Center  receives  requests  from  the  field  for  technical  assistance,  which  it  then  fills  in-house  or 
refers  to  other  technical  assistance  providers.  In  fiscal  year  2000,  it  either  filled  or  referred  329 
requests  for  technical  assistance.  The  Center  does  not  provide  direct  training  to  the  field. 

‘'Not  applicable. 

“Data  reported  from  19  other  training  and  technical  assistance  grantees. 

'Data  not  available. 

^Reflects  the  number  of  communities  that  received  technical  assistance. 
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"At  the  suggestion  of  OJJDP  officials  responsible  for  the  following  programs,  data  for  the  Community 
Prevention  Grants  Program  are  prorated  from  the  first  9  months  of  the  fiscal  year,  training  and 
technical  assistance  data  for  State  Challenge  Activities  are  prorated  from  the  first  8  months  of  the 
fiscal  year,  and  data  for  the  Comprehensive  Strategy  for  Serious,  Violent,  and  chronic  Juvenile 
Offenders  are  prorated  from  the  first  6  months  of  the  fiscal  year. 

includes  data  from  the  national  grantee  and  three  of  the  four  regional  grantees. 

'Data  are  for  fiscal  year  2000. 

Source:  GAO  analysis  of  OJJDP  and  grantee  data. 

OJJDP  officials  cautioned  that  not  all  providers  share  common  definitions 
of  “training”  and  “technical  assistance.”®  For  one  thing,  the  difference 
between  the  two  is  not  always  clear  and,  therefore,  it  is  sometimes 
difficult  to  definitively  categorize  a  provided  service  as  training  versus 
technical  assistance.  Furthermore,  not  all  training  events  are  equal.  For 
example,  some  providers  might  characterize  both  a  1-day  training 
conference  and  a  10-day  training  workshop  as  a  training  event;  others 
might  differentiate  between  the  two.  Furthermore,  one  provider  might 
consider  a  telephone  request  from  a  grantee  as  merely  a  query,  while 
another  might  consider  it  a  request  for  technical  assistance. 


Pprf nrm  n  pp  f nr  OJJDP  administers  its  research  grants  out  of  its  Research  and  Program 

Development  Division  (RPDD).^  RPDD  sponsors  empirical  studies  on  an 
R6S6arch  Grant66S  array  of  topics  related  to  juverules  and  delinquency,  firom  the  causes  of 

violence  to  the  impact  of  victimization.  The  overall  goal  of  these  research 
grants  is  to  generate  credible  and  useful  information  to  help  prevent  and 
reduce  juvenile  delinquency  and  victimization.  Research  grantees  are  not 
only  expected  to  collect  data  but  to  analyze  and  disseminate  their  analyses 
to  the  public.  RPDD  requires  aU  research  grantees  to  produce  publishable 
products  and,  in  some  instances,  RPDD  specifies  the  type  of  products  to 
be  published  depending  on  the  results  of  the  research.  Thus,  according  to 
OJJDP  officials,  one  measure  of  a  research  grantee’s  performance  is  the 
number  of  products  the  grantee  has  published. 

Like  most  OJJDP  grantees,  research  grantees  must  report  information  on 
their  projects’  activities  and  accomplishments  semiannually  through 


®  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  pointed  out  that 
TTAD  published  the  Training,  Technical  Assistance,  and  Evaluation  Protocols  booklet  2 
years  ago,  which  defined  both  training  and  technical  assistance. 

^  Some  grants  administered  through  other  OJJDP  divisions  have  a  research  component,  but 
they  are  not  considered  “research  grants.” 
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progress  reports.  RPDD  does  not  impose  additional,  specific  reporting 
requirements  on  grantees,  but  it  does  encourage  them  to  report  on 
products  produced  through  private  publishers  (as  opposed  to  those 
published  through  OJJDP).^^  The  division  director  told  us  that  it  is  not 
necessary  to  impose  specific  requirements  on  grantees  in  addition  to  the 
semiannual  progress  report  requirements  because  officials  work  closely 
with  grantees  throughout  the  life  of  the  grants. 

OJJDP  research  grantees  produce  products  based  on  their  OJJDP-funded 
research.  Some  of  these  products  are  approved  and  published  by  OJJDP, 
who  in  turn  disseminates  the  products  through  its  own  distribution 
process  (see  app.  n  for  a  description  of  OJJDP’s  product  dissemination 
process).  Grantees  also  publish  many  products  that  are  based  on  their 
OJJDP-funded  research  through  private  publishers.  OJJDP  officials  told  us 
they  give  their  research  grantees  latitude  to  privately  publish  products 
because  the  majority  of  their  research  grantees  are  academics  whose 
funding  depends  on  the  number  of  products  they  publish,  and  because 
grantees  often  have  funding  sources  in  addition  to  OJJDP. 

Tables  10  and  11  summarize  the  products  that  active  OJJDP  research 
grantees  published  through  OJJDP  and  private  publishers  as  a  direct  result 
of  OJJDP-funded  research.  Table  10  describes  the  number  of  research 
products  published  by  OJJDP  firom  1993  through  September  2000,  by 
topic.  Table  11  shows  the  number  of  products,  by  topic,  that  grantees  with 
active  research  grants  privately  published  between  1986  and  Jime  2001,  or 
were  in  the  process  of  publishing  in  June  2001. 


Officials  told  us  that  they  ask  grantees  to  submit  products  for  review  prior  to  external 
publication,  but  often  grantees  do  not  do  this.  In  the  past,  grant  applicants  have  turned 
down  funding  when  it  was  contingent  on  OJJDP’s  review  of  privately  published  products, 
citing  the  importance  of  academic  freedom.  Also,  grantees  oppose  submitting  external 
products  to  OJJDP  prior  to  publication  because  many  academic  journals  will  not  publish 
research  findings  that  have  been  reviewed  or  disseminated  prior  to  submission. 
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Table  10:  Active  OJJDP  Grantees’  Research  Products,  Published  by  OJJDP 


Topic 

Summaries 

Reports 

Fact  sheets 

Bulletins 

Aftercare  (post-detention) 

2 

1 

1 

3 

Causes  and  Correlates  of 
Delinquency® 

1 

2 

4 

6 

Crimes  against  juveniles 

0 

0 

0 

7 

Delinquency  prevention 

1 

5 

1 

6 

Detention/corrections 

3 

2 

7 

2 

Gangs 

5 

0 

4 

8 

Juvenile  courts 

1 

3 

9 

4 

Juvenile  health 

0 

1 

1 

2 

Juvenile  justice  system  reform 

1 

2 

0 

2 

Juvenile  offenders 

3 

3 

5 

10 

School  safety 

0 

2 

0 

0 

Other 

0 

1 

1 

0 

Total 

17 

22 

33 

50 

Note:  Table  Includes  some  products  published  as  a  result  of  OJJDP  evaluation  and  statistics  grants, 

^Thls  topic  Is  comprised  of  products  from  Causes  and  Correlates  of  Delinquency  grants,  in 
commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  noted  that  these  products  can  be 
categorized  under  a  number  of  the  topics  listed. 

Source:  GAO  analysis  of  OJJDP  data. 


Table  11:  Active  OJJDP  Grantees’  Research  Products,  Published  Externally 

Topic 

Journal 

articles 

Book  chapters 

other 

publications* 

Causes  and  Correlates  of 
Delinquency*’ 

57 

43 

17 

Crimes  against  juveniles 

12*= 

6 

2 

Delinquency  prevention 

0 

0 

2 

Detention/corrections 

3 

0 

0 

Gangs 

13** 

9 

2 

Juvenile  courts 

5 

1 

1 

Juvenile  health 

51® 

5 

7 

Juvenile  offenders 

3 

0 

2 

School  safety 

16 

0 

4 

Other 

0 

0 

5 

Total 

160 

64 

42 

Note:  Table  includes  some  products  published  as  a  result  of  OJJDP  evaluation  and  statistics  grants. 
“Includes  books,  reports,  and  other  publications. 

‘This  topic  is  comprised  of  products  from  Causes  and  Correlates  of  Delinquency  grants.  In 
commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  noted  that  these  products  can  be 
categorized  under  a  number  of  the  topics  listed. 
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^Includes  one  paper  submitted  but  not  yet  accepted  for  publication. 
“Includes  one  paper  presented  at  an  academic  conference. 

“Includes  three  papers  submitted  but  not  yet  accepted  for  publication. 
Source:  GAO  analysis  of  OJJDP  and  grantee  data. 
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Table  12;  State-Reported  Compliance  with  Core  Requirements,  Calendar  Year  1998 

Core  requirements 

Number  of 
jurisdictions 

Jurisdictions* 

Deinstitutionalization  of  status  offenders 

Full  compliance  -  zero  violations 

9 

ME,  MT,  NY,  VT,  AS,  GU,  MP,  PR,  VI 

Full  compliance  -  de  minimis  exceptions^" 

39 

AL,  AK,  AR,  CA,  CO,  CT,  DE,  DC,  FL,  GA,  HI,  ID,  IL,  IN, 
lA,  KS,  KY,  LA,  MD,  MA,  MN,  MO,  NE,  NV,  NH,  NM,  NC, 

ND.  OK,  OR.  PA,  Rl.  TX.  UT,  VA.  WV,  Wl 

Not  In  compliance 

4 

MS,  SC,  TN,  WA 

Funds  withheld  pending  additional  compliance  data 

2 

NJ,OH 

Not  participating 

2 

SD,WY 

Separation  of  iuvenile  and  adult  offenders 

Full  compliance  -  zero  violations 

41 

AL.  AZ,  AR.  CA,  CO,  CT,  DE,  DC,  FL,  IL,  IN,  KS,  KY,  ME, 
MD,  MA,  Ml,  MN,  MO,  MT,  NE,  NV,  NH,  NM,  NC,  ND,  OK, 
OR.  PN,  Rl,  SC,  UT,  VT,  VA,  WA,  Wl,  AS.  GU,  MP,  PR,  VI 

Full  compliance  -  exception  provision 

11 

AL,  GA,  HI,  ID,  lA,  LA,  MS,  NY,  TN,  TX,  WV 

Funds  withheld  pending  additional  compliance  data 

2 

NJ,  OH 

Not  participating 

2 

SD,  WY 

Jail  and  lockup  removal 

Full  compliance  -  zero  violations 

11 

AL,  DC,  NC,  OR,  SC,  VT,  AS,  GU,  MP,  PR,  VI 

Full  compliance  -  de  minimis  exceptions 

38 

AZ,  AR,  CA,  CO,  CT,  DE,  FL,  GA,  HI,  ID,  IN.  lA,  KS,  KY, 

LA,  MD,  MA,  Ml.  MN,  MS,  MO,  MT,  NE,  NV,  NH,  NM,  NY, 
ND,  OK,  PA,  Rl,  TN,  TX,  UT,  VA.  WA,  WV,  Wl 

Not  in  compliance 

3 

AK,  IL,  ME 

Funds  withheld  pending  additional  compliance  data 

2 

NJ,  OH 

Not  participating 

2 

SD,  WY 

Disproportionate  minority  confinemenf 

Completed  identification  and  assessment, 
implementing  Intervention  and  monitoring 

21 

AR,  CO,  CT,  FL,  GA,  ID,  IN,  lA,  LA.  MO,  NV,  NC,  ND,  OK, 
PA,  SC,  TN,  UT,  VA,  WA,  WV 

Completed  identification  and  assessment, 
implementing  Intervention 

15 

AK,  AZ,  HI,  IL,  KS,  MD,  MA,  Ml,  MN,  NE,  NM,  NY,  Rl,  TX, 

Wl _ 

Completed  identification  and  assessment/updating 
data  and  revising  intervention  plan 

2 

CA,  DE 

Completed  identification/implementing 
intervention/conducting  formal  assessment 

3 

AL,  MS,  MT 

Conducting  identification  phase 

1 

NH 

Exempt  from  requirement 

7 

ME,  VT,  AS,  GU,  MP.  PR,  VI 

Status  under  review 

5 

DC,  KY,  NJ,  OH,  OR 

Not  participating 

2 

SD,  WY 

Note:  OJJDP  based  its  initial  determination  of  states’  eligibility  to  receive  fiscal  year  2000  Formula 
Grants  on  1998  monitoring  reports — the  most  recent  data  available  as  of  March  2001. 


®AS  =  American  Samoa;  GU  Guam;  MP  =  Northern  Mariana  Islands;  VI  =  Virgin  Islands. 

"Fewer  than  29.4  violations  per  10,000  persons  under  age  18  in  the  state. 

"States  reported  data  on  disproportionate  minority  confinement  in  their  fiscal  year  2000 
comprehensive  plans  for  compliance. 
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This  appendix  contains  information  on  the  10  impact  evaluations  that  the 
Office  of  Juvenile  Justice  and  Delinquency  Prevention  (OJJDP)  has  funded 
of  its  own  programs  since  1995  and  for  which  we  have  assessed  the 
methodological  rigor,  as  well  as  information  on  one  impact  evaluation — 
Teen  Courts — ^that  we  did  not  assess.  Five  of  the  10  evaluations  are  in  their 
formative  stages,  and  five  are  well  into  their  implementation.  For  each  of 
the  10,  we  have  included  a  description  of  the  program  being  assessed,  the 
evaluating  organization,  a  description  of  the  evaluation  and  its  findings, 
and  our  assessment  of  the  evaluation.^  As  discussed  in  the  Scope  and 
Methodology  section  of  this  report,  we  did  not  assess  the  methodological 
rigor  of  the  Teen  Courts  evaluation.  However,  we  have  included  a 
summary  of  this  evaluation  at  the  end  of  this  appendix.  ^ 

Impact  Evaluations  of 

OJJDP  Programs  in 

Their  Formative 

Stages 

Parents  Anonymous  Program  Description'.  Parents  Anonymous  is  a  national  child  abuse 

prevention  program  that  began  in  1970.  It  consists  of  32  state  and  local 
organizations  and  over  1,000  weekly  mutual  support  groups.  The  principal 
participants  are  at-risk  parents,  though  complementary  projects  exist  for 
children.  The  cornerstones  of  the  program  are  mutual  support  and  shared 
leadership. 

Evaluator:  National  Council  on  Crime  and  Delinquency. 

Evaluation  description:  This  evaluation,  which  is  in  the  beginning 
stages,  is  based  on  a  proposal  to  conduct  a  process  evaluation  in  year  1, 
and  an  outcome  evaluation  in  years  2  and  3.  The  researchers  will 
determine  how  Parents  Anonymous  is  staffed  and  operated  in  different 
settings,  how  it  attempts  to  change  the  behavior  and  attitudes  of  parents, 
and  what  factors  are  related  to  its  effectiveness.  While  the  specifics  of  an 
outcome  evaluation  design  are  yet  to  be  determined,  the  researchers 


'  We  relied  on  documents  OJJDP  provided  to  us  in  April  2001  in  assessing  the  evaluation 
methodologies  and  reporting  on  each  evaluation’s  status. 

^  OJJDP  provided  us  with  this  evaluation  description. 
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indicate  that  they  will  most  likely  compare  the  Parents  Anonymous 
participants  with  a  control  group  and  with  Parents  Anonymous  dropouts. 
The  process  evaluation  received  $300,000  for  a  3-year  period. 

Evaluation  findings:  It  is  too  early  in  this  evaluation  to  have  reported 
results. 

GAO  assessment:  No  assessment  of  the  impact  evaluation  is  possible 
because  it  has  not  yet  been  planned.  More  fully  developed  proposals  wiU 
need  to  be  made  to  OJJDP  to  obtain  funding  for  the  impact  evaluation 
portion  of  the  study. 


Positive  Action  Through 
Holistic  Education 
(Project  PATHE) 


Program  Description:  This  program  replicates  and  evaluates  Project 
PATHE,  which  was  first  implemented  in  the  Charleston  County  School 
District  in  South  Carolina  between  1980  and  1983.  Project  PATHE  is  a 
comprehensive  school-based  program  that  combines  services  to  students 
who  are  at  elevated  risk  for  developing  problem  behaviors  with  school¬ 
wide  organizational  changes  intended  to  improve  both  school  climate  and 
students’  behavior.  Local  educators  are  encouraged  to  develop  their  own 
(1)  explanations  for  the  causes  of  their  schools’  violence  and  behavior 
problems  and  (2)  specific  local  objectives  and  ways  to  prevent  these 
problems  by  empowering  teachers’  decision-making  and  fostering 
collaborative  and  nonhierarchical  efforts.  For  various  reasons,  the  grantee 
has  had  difficulty  in  selecting  a  school  district  for  the  replication.  Funding 
for  this  effort  began  in  October  1999  with  funds  provided  by  the  Centers 
for  Disease  Control  and  Prevention. 


In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General 
pointed  out  that  all  funds  for  this  effort  come  from  the  Centers  for  Disease 
Control  and  Prevention  through  an  interagency  agreement.  OJJDP 
awarded  this  grant,  which  the  two  agencies  jointly  manage.  OJJDP 
originally  identified  Project  PATHE  as  an  OJJDP-funded  program. 
However,  on  the  basis  of  the  Assistant  Attorney  General’s  comments,  this 
program  appears  to  be  a  non-OJJDP-funded  impact  evaluation.  Since  we 
assessed  its  methodological  rigor,  we  have  included  it  with  the  other 
OJJDP-fimded  evaluations. 


Evaluator:  Institute  of  Behavioral  Science,  University  of  Colorado. 

Evaluation  description:  Although  the  impact  evaluation  was  expected 
to  be  completed  in  Jidy  2001,  it  has  yet  to  begin.  Ori^al  plans  called  for 
one  school  district  to  be  selected  for  the  replication  and  evaluation.  In  a 
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school  district,  one  high  school  and  one  middle  school  ’would  be  selected 
to  receive  the  program,  once  school  principals  had  been  informed  and 
staff  surveys  had  been  conducted  to  determine  interest  in  participating. 
Comparison  schools  (one  high  school  and  one  middle  school)  in  the  same 
district  would  be  selected  with  similar  demographic  characteristics  of 
students,  levels  of  problem  behaviors,  and  other  unspecified 
organizational  characteristics,  as  well  as  a  low  probability  of  mounting 
other  school-wide  efforts  to  reduce  problem  behavior  during  the  study 
period.  Plans  are  to  collect  data  before  and  after  project  implementation. 
AU  students  and  staff  in  all  schools  would  be  surveyed  in  September  and 
May  for  3  consecutive  school  years.  In  addition,  200  students  from  each 
high  school — 100  seniors  and  100  sophomores — and  200  students  from 
each  middle  school — 100  eighth  graders  and  100  sixth  graders — ^would  be 
sampled  in  the  first  year  of  the  study  for  follovmp  for  3  years.  Each  sample 
is  to  include  25  students  identified  as  “high  risk.”  It  is  not  clear,  however, 
how  these  samples  will  be  selected.  Schools  are  to  be  visited  three  times 
yearly  during  the  study,  and  school  records  and  teacher  ratings  of  student 
behavior  in  each  school  will  be  used  in  establishing  differences  between 
the  program  and  comparison  schools.  Program  outcomes  are  to  be 
selected  after  determining  goals  and  objectives  collaboratively  with  local 
school  officials.  Multivariate  statistical  analyses,  such  as  logistic 
regression,  are  planned.  Because  of  the  difficulty  in  selecting  a  school 
district  the  latest  progress  report  indicates  some  changes  in  this  design. 
Agreements  have  been  signed  with  two  school  districts  (Charleston,  SC 
and  Baltimore,  MD),  instead  of  one,  and  plans  are  to  conduct  the 
replication  and  evaluation  in  four  middle  schools  (two  program  and  two 
comparison  schools)  in  each  district — high  schools  have  been  excluded. 
The  amount  of  the  grant  is  $875,000.  However,  additional  funds  have  been 
requested. 

Evaluation  findings:  It  is  too  early  in  this  evaluation  to  have  reported 
results. 

GAO  assessment:  The  evaluation,  as  designed,  is  basically  sound.  The 
variation  in  program  structure  and  implementation  between  schools  may 
limit  generalizability.  While  the  researchers  do  suggest  awareness  of 
potential  problems  due  to  students  switching  schools,  they  do  not  clearly 
indicate,  at  this  early  design  stage,  how  possible  contamination  wUl  be 
handled. 


Rural  Gang  Initiative  Program  Description:  The  Rural  Gang  Initiative  is  a  comprehensive 

strategy  to  ameliorate  gang  problems  in  rural  areas.  The  program  was 
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adapted  from  the  comprehensive  gang  model,  developed  at  the  University 
of  Chicago,  and  implementation  began  in  two  rural  areas  in  fall  2000.  The 
program  consists  of  five  elements:  community  mobilization,  opportumties 
provision,  social  intervention,  suppression,  and  organizational  change  and 
development. 

Evaluator:  National  Council  on  Crime  and  Delinquency. 

Evaluation  description:  The  impact  evaluation  of  this  program  began  in 
January  2001  and  is  expected  to  be  completed  in  December  2003.  It 
focuses  on  two  of  the  four  sites,  Mount  Vernon  (IL)  and  Glenn  County 
(CA),  that  were  part  of  a  year-long  feasibility  study  that  began  in  April 
1999.  The  other  two  sites  are  not  part  of  the  evaluation  because  they  did 
not  fully  implement  the  model.^  Information  will  be  collected  from  these 
two  sites  on  gang-involved  youths  and  youths  at-risk  of  gang  involvement, 
all  of  whom  have  participated  in  the  program.  However,  it  is  unclear  how 
the  youths  will  be  sampled  or  whether  all  participants  will  be  included. 
Data  will  be  obtained  firom  a  variety  of  sources,  including  interviews, 
organizational  surveys,  and  administrative  data  fi:*om  schools  and  the 
justice  system.  No  comparison  groups  are  planned,  though  the  researchers 
indicate  that  attempts  wiQ  be  made  to  collect  data  that  will  permit  an 
assessment  of  alternative  explanations  for  program  effects.  The 
researchers  plan  to  collect  data  before  and  after  program  implementation; 
then  they  will  measure  any  changes  and  follow-up  Avith  program 
participants  for  at  least  12  months  after  their  participation  in  the  program. 
Individual  sites  are  to  identify  specific  program  outcomes.  Although  the 
researchers  have  described  prospective  outcome  measures,  such  as  the 
reduction  of  gang-related  crime  and  the  prevention  or  reduction  of  gang 
involvement,  they  (1)  have  not  chosen  the  outcome  measures  that  they 
vriU  use  and  (2)  have  not  provided  information  on  the  types  of  statistical 
analyses  planned.  However,  the  evaluation  is  stiU  in  the  formative  stages. 
At  the  time  of  our  review,  this  evaluation  had  received  $525,000  in  funding. 

Evaluation  findings:  It  is  too  early  in  this  evaluation  to  have  reported 
results. 


^  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  pointed  out  that 
these  two  sites  did  not  implement  the  comprehensive  gang  model  because  their  gang  crime 
problem  did  not  warrant  the  modefs  prescribed  intervention  approach. 
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GAO  assessment:  It  is  too  early  to  teU  from  this  evaluation  how  effective 
it  will  be.  However,  the  absence  of  plans  for  comparison  groups  and  the 
lack  of  specificity  regarding  any  other  control  mechanisms,  make  it 
unclear  how  program  effects  will  be  distinguished  from  alternative 
explanations  at  this  stage  of  the  evaluation. 


Safe  Schools/Healthy  Program  Description:  The  Safe  Schools/Healthy  Students  program  has 

Students  funded  77  school  districts  nationwide,  with  grants  ranging  up  to  $3  million, 

to  develop  services  and  activities  to  promote  healthy  childhood 
development  and  prevent  violence  and  drug  abuse.  The  program  also  aims 
to  develop  greater  collaboration  and  cooperation  between  communities 
and  schools  to  enhance  their  effectiveness  in  responding  to  and  reducing 
violence.  Each  project  model  is  intended  to  evolve  over  time. 

Evaluator:  Research  Triangle  Institute. 

Evaluation  description:  This  impact  evaluation  began  in  October  1999 
and  data  will  be  collected  through  the  Spring  of  2005.  School  and 
community-based  archival  records,  surveys  of  key  coalition  personnel, 
teachers,  superintendents,  principals  and  other  school  staff,  and  teacher 
behavioral  checklists  for  students  in  selected  grades  will  be  gathered  in  all 
77  school  districts.  The  evaluation  will  compare  data  from  participating 
sites  with  national  norms  and  with  similar  information  from  matched 
nonparticipating  (comparison)  sites  that  the  researchers  surveyed  in  each 
of  two  large,  nationally  representative  studies  of  school  districts.  The 
matching  will  be  based  on  unspecified  socio-demographic  characteristics 
and  responses  to  policy-related  questions  in  the  baseline  survey.  Archival 
data  will  be  collected  yearly  over  a  5-year  period.  Survey  data  will 
generally  be  collected  at  three  points  in  time,  about  2  years  apart.  The 
survey  items  will  be  drawn  from  established  instruments  and  will  provide, 
in  conjunction  with  the  archival  or  administrative  data,  information  on 
behavioral  outcomes,  risk  factors  and  inhibiting  factors,  and  indicators  of 
positive  development  and  mental  health.  At  the  time  of  our  review,  this 
evaluation  was  funded  at  approximately  $5.6  million. 

Evaluation  findings:  It  is  too  early  in  this  evaluation  to  have  reported 
results. 

GAO  assessment:  This  evaluation,  as  designed,  is  basically  sound. 
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Safe  Start  Initiative 


Program  Description:  This  demonstration  program  seeks  to  prevent  and 
reduce  the  impact  of  family  and  community  violence  on  yoimg  children, 
primarily  aged  0  to  6,  in  up  to  12  communities.  The  program  plans  to 
create  a  comprehensive  service  delivery  system  that  integrates  service 
providers  (in  the  fields  of  early  childhood  education/development,  health, 
mental  he^th,  and  all  manner  of  prevention,  intervention,  and  treatment 
programs),  law  enforcement,  legal  services,  and  the  courts.  It  also  seeks  to 
improve  the  access,  delivery,  and  quality  of  services  to  children  exposed 
to,  and  at  high  risk  of,  violence.  Project  sites  are  to  be  selected  through  a 
competitive  grant  process.  Funding  for  the  program  began  in  October 
1999. 

Evaluator:  Caliber  Associates. 

Evaluation  description:  This  evaluation  began  in  May  2000,  and  is 
expected  to  end  in  September  of  2005.  The  effect  of  the  Safe  Start 
Initiative  will  be  measured  within  and  across  all  participating  commumties 
at  both  the  community  and  individual  levels.  Multiple  data  collection 
methods,  including  focus  groups,  service  agency  usage  logs  and 
documents,  and  random-digit-dialing  telephone  surveys,  will  be  used. 

Plans  are  to  collect  data  before  program  implementation  and  for  4  years 
after  the  program  begins  in  each  site.  Although  specific  outcome  measures 
have  not  yet  been  identified,  they  are  expected  to  address  such  areas  as 
agency  referral  levels  and  quality  of  service,  increased  interagency 
collaboration,  knowledge  and  perceptions  of  police  and  child  protective 
services,  rates  of  child  maltreatment,  physical  iryuries,  and  mental  health 
problems.  No  comparison  commimities  are  to  be  studied.  Analyses  are  to 
include  regression  and  time  series  models.  At  the  time  of  our  review,  this 
initiative  had  received  $1  million. 

Evaluation  findings:  It  is  too  early  in  this  evaluation  to  have  reported 
results. 

GAO  assessment:  The  absence  of  any  appropriate  comparison 
communities  and  the  variability  in  program  implementation  and 
components  across  the  12  study  sites  will  make  it  difficult  to  find 
compelling  evidence  of  program  effects. 
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Impact  Evaluations  of 
OJJDP  Programs  Well 
Into  Their 
Implementation 

Program  Description:  This  program  aims,  in  the  five  sites  in  which  it  is 
being  implemented  and  evaluated  (Mesa  and  Tucson,  AZ;  Bloomington,  IL; 
San  Antonio,  TX;  and  Riverside,  CA)  to  reduce  gang-related  crime  through 
five  interrelated  strategies:  community  mobilization;  provision  of  social, 
educational,  and  economic  opportunities;  suppression  of  gang  violence; 
social  intervention;  and  organizational  innovation.  It  involves  the 
collaborative  efforts  of  the  police,  probation  officers,  prosecutors,  judges, 
schools,  youth  agencies,  churches,  housing  authorities,  and  governmental 
agencies.  It  targets  youths  at  strong  risk  of  gang  membership  and  crime, 
and  youths  already  involved  in  serious  gang  crime.  The  evaluation  began 
in  1995. 


Comprehensive 
Communitywide  Approach 
to  Gang  Prevention, 
Intervention,  and 
Suppression  Program 
(Comprehensive  Gang 
Initiative) 


Evaluator:  University  of  Chicago. 

Evaluation  description:  The  impact  evaluation  of  the  program  began  in 
May  1995,  and  is  expected  to  be  completed  in  April  2002.  Each  project  site 
is  to  be  matched  with  a  comparison  site.  In  four  of  the  five  sites,  the 
program  participants  and  comparison  groups  were  selected  from  similar 
gang  problem  areas  within  the  same  city;  in  the  fifth  site,  a  separate 
comparison  community  was  selected.  Between  100  and  115  youths,  ages 
12  to  21,  who  were  involved  in  gangs  or  at  risk  of  involvement,  were 
selected  to  participate  in  the  program  in  each  site,  and  between  77  and  134 
similar  youths  in  each  site  were  selected  for  comparison  purposes.  Neither 
program  nor  comparison  group  youths  were  selected  randomly.  A  large, 
complex,  communitywide  data  collection  effort  is  being  employed  in  each 
site,  through  a  variety  of  methods  and  sources,  including  organizational 
surveys,  youth  surveys,  reports  from  service  workers,  police  and  school 
records,  local  newspaper  reports,  and  census  data.  Data  were  to  be 
collected  at  baseline  and  after  the  first  and  third  years  of  the  program.  The 
principal  outcomes  to  be  measured  are  gang  crime  patterns  at  the 
individual,  gang,  and  community  levels.  The  evaluation  will  also  consider 
changes  in  opportunities,  as  weU  as  integration  in  and  alienation  from 
conventional  individuals  and  institutions.  A  variety  of  analyses  of  the  data 
are  planned,  including  time-series  analyses  and  hierarchical  linear  models. 
At  the  time  of  our  review,  the  evaluation  had  received  approximately  $3 
million. 


Page  62 


GAO-02'23  OJJDP  Grantee  Reporting  Requirements  and  Concerns 


Appendix  VII;  Impact  Evaluations  OJJBP  Has 
Funded  of  Its  Own  Programs 


Evaluation  findings:  Preliminary  results  have  been  reported  for 
program  participants,  but  not  comparison  groups.  Because  of  missing  data 
and  other  problems,  reporting  deadlines  may  not  be  met,  and  two  of  the 
five  project  sites  and  associated  comparison  sites  have  been  deferred  from 
the  current  analyses. 

GAO  assessment:  The  evaluation,  as  designed,  is  basically  soimd. 
However,  numerous  difficulties  in  obtaining  data  threaten  parts  of  the 
evaluation.  Also,  the  way  in  which  subjects  were  selected  for  the  study 
may  be  problematic,  and  it  is  unclear  how  program  and  comparison 
youths  were  matched  within  sites. 


Enforcing  the  Underage  Program  Description:  Since  1998, 85  communities  and  4  colleges  in  at 
Drinking  Laws  Program  least  10  states  have  been  awarded  subgrants  under  the  discretionary  grant 

component  of  this  program  to  enforce  underage  drinking  laws.  In  most 
states,  a  diverse  group  of  stakeholders  are  involved  in  planning  a  variety  of 
projects  under  this  program  that  can  include  media  campaigns,  merchant 
education,  compliance  checks  and  other  enforcement,  youth  leadership 
training,  school-based  education,  and  the  development  of  local  coalitions 
and  interventions  aimed  at  reducing  underage  drinking.  States  and 
communities  are  given  substantial  latitude  in  planning  their  projects; 
interventions  are  not  standardized  across  communities. 

Evaluator:  Wake  Forest  University  School  of  Medicine. 

Evaluation  description:  The  effort  to  evaluate  the  discretionary  grant 
component  of  this  program  began  October  1, 1998,  and  is  expected  to  be 
completed  December  31, 2001.  Data  are  to  be  collected — ^from  telephone 
surveys  of  police  chiefs,  sheriffs,  and  youths  in  participating  communities 
and  matched  comparison  communities  in  at  least  9  states — ^before  or  early 
on  in  project  implementation  and  at  least  1  year  after  project  initiation. 
Project  sites  to  be  evaluated  were  initially  chosen  from  all  states 
participating  in  the  program.  It  is  unclear  whether  these  sites  are 
representative  of  all  participating  project  sites.  In  the  first  year,  surveys 
were  conducted  in  52  participating  communities  and  a  sitmlar  number  of 
comparison  communities.  In  the  second  and  third  years,  surveys  will  be 
conducted  in  those  same  communities  and  34  others — 17  in  each  group. 
The  participant  and  comparison  communities  were  matched  on  median 
income,  liquor  law  violations,  percentage  attending  college,  and 
population  size.  The  surveys  of  the  top  one  or  two  law  enforcement 
officials  in  each  community  wiQ  provide  information  on  local  law 
enforcement  efforts,  including  the  number  of  compliance  checks 
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conducted  in  each  year.  A  small  number  of  youths  from  each  site  are  to  be 
selected  at  random  for  the  surveys  each  year.  The  youth  surveys  will 
obtain  data  on  perceptions  of  alcohol  availability,  peer  and  personal 
alcohol  use,  and  alcohol-related  problem  behaviors  including  binge 
drinking  and  drunk  driving.  At  the  time  of  our  review,  this  evaluation  had 
received  approximately  $945,000. 

Evaluation  findings:  While  some  demographic  data  have  been  reported 
from  the  baseline  survey,  no  results  have  been  reported  involving  program 
effects. 

GAO  assessment:  The  researchers  suggest  aggregating  all  program 
communities  together  and  all  comparison  communities  together  to 
diminish  community  sample  size  problems,  which  may  mask  program 
effects,  hi  addition,  the  wide  variation  allowed  in  program  implementation 
may  compromise  the  interpretation  and  generahzability  of  any  findings. 


Intensive  Aftercare  Program  Description:  The  Intensive  Aftercare  program  provides 

intensive  supervision  and  services  to  serious  juveiule  offenders  for  6 
months  following  their  release  from  secure  confinement.  The  goal  is  to 
facilitate  reintegration  and  reduce  recidivism.  The  program  was 
implemented,  beginning  in  June  of  1993,  by  various  youth  service  offices 
and  departments  of  corrections  in  four  states:  Colorado,  Nevada,  New 
Jersey,  and  Virginia.  New  Jersey  was  eventually  dropped  because  of 
implementation  problems,  so  the  evaluation  of  the  program  is  being 
completed  in  the  other  three  states. 

Evaluator:  National  Council  on  Crime  and  Delinquency. 

Evaluation  description:  Beginning  in  1995,  youths  entering  correctional 
facilities  in  the  three  states  (four  counties  in  Colorado  including 
Metropohtan  Denver;  Clark  County,  NV;  and  Norfolk  Coimty,  VA)  were 
screened  for  eligibility  and  randomly  assigned,  within  each  site,  to  the 
treatment  group  (whose  members  participated  in  the  Intensive  Aftercare 
program  upon  release)  or  control  group.  Between  1995  and  1999, 82 
youths  were  assigned  to  the  program  and  68  to  the  control  group  in 
Colorado,  120  youths  were  assigned  to  the  program  and  127  to  the  control 
group  in  Nevada,  and  75  youths  were  assigned  to  the  program  and  45  to 
the  control  group  in  Virginia.  Information  was  collected  for  study 
participants  at  baseline  (that  is,  upon  entry  into  the  institution),  before 
release  from  the  institution  (9  to  12  months  after  baseline,  or  entry), 
immediately  after  completing  the  program  (6  months  after  release),  and  6 
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months  after  completing  the  program  (12  months  after  release).  The  data 
collected,  using  survey  instruments,  standardized  tests,  monthly  case 
management  forms,  and  administrative  (police  and  court)  databases, 
included  social  and  criminal  history  and  demographic  data,  information  on 
the  extent  of  supervision  and  services,  and  the  extent  of  criminal  activity 
following  institutional  release.  Many  of  the  measures  being  employed  in 
the  study,  according  to  the  researchers,  are  standard  and  have  been 
validated.  At  the  time  of  our  review,  the  evaluation  had  received 
approximately  $932,000  in  funding. 

Evaluation  findings:  The  preliminary  findings  offered  from  this 
evaluation  suggest  that  the  Intensive  Aftercare  participants  did  receive 
greater  supervision  and  more  services  after  release  than  the  control  group, 
which  suggests  some  success  in  implementing  the  program.  Outcome 
results  related  to  reintegration  and  recidivism  are  not  complete,  and  the 
interim  results  are  mixed  as  to  whether  the  program  is  associated  with 
positive  outcomes. 

GAO  assessment:  This  is  a  well-designed  study,  though  serious  missing 
data  problems,  if  not  corrected,  may  make  it  difficult  to  determine  the 
outcome  of  this  program. 


Juvenile  Mentoring 
Program  (JUMP) 


Program  Description:  JUMP  was  established  by  Part  G  of  the  Juvenile 
Justice  and  Delinquency  Prevention  Act  of  1974,  as  amended  in  1992. 
Through  that  legislation,  the  Congress  authorized  OJJDP  to  award  3-year 
grants  to  community-based  not-for-profit  organizations  or  to  local 
educational  agencies.  The  grantees  are  to  support  one-on-one  mentoring 
projects  that  match  volunteer  adult  mentors  with  youths  at  risk  of 
delinquency,  gang  involvement,  educational  failure,  and  dropping  out  of 
school.  The  legislation  also  provided  funding  for  a  national,  cross-site 
evaluation  of  JUMP.  OJJDP  guidelines  emphasize  the  need  for  projects  to 
recruit,  train,  supervise,  and  do  thorough  background  checks  for  all 
volunteer  mentors;  develop  procedures  for  appropriately  matching  youths 
and  mentors;  define  the  population  of  at-risk  youths  to  be  served;  develop 
guidelines  for  the  type,  frequency,  and  duration  of  youth  and  mentor 
project  activities;  and  establish  procedures  for  gathering  and  reporting 
data  to  support  the  evaluation  process.  As  of  November  2000, 175  JUMP 
projects  had  been  funded,  in  amounts  ranging  from  $180,000  to  $210,000 
over  a  3-year  period. 

Evaluators:  Information  Technology  International  and  Pacific  Institute 
for  Research  and  Evaluation. 
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Evaluation  description:  This  evaluation  began  in  May  1997  and  is 
expected  to  conclude  on  September  30,  2002/  Three  approaches  are  being 
taken  to  determine  how  weU  JUMP  is  accomplishing  its  objectives.  The 
first  is  a  modified  pre-post  design  that  involves  a  within-subject 
comparison  of  the  characteristics  of  youths  at  the  time  they  enter  and  exit 
the  program  and  between-subject  comparisons  of  youths  entering  and 
exiting  the  program  at  the  same  time.  The  second  approach  is  a  best 
practices  approach  that  will  use  structural  equation  models  to  estimate 
what  program  features  or  activities,  including  success  in  matching 
mentors  and  youths,  are  most  likely  to  contribute  to  program  success  in 
reducing  the  risk  of  school  and  family  problems,  delinquency,  and  drug 
use  among  youths.  The  third  approach  relies  on  combined  youth  outcome 
data  and  community  data  to  determine  community  cost  offsets.  The 
evaluation  was  funded  at  $3.3  million. 

Evaluation  findings:  In  their  November  2000  JUMP  armual  report,  the 
evaluators  provided  considerable  descriptive  information  about  the 
various  JUMP  projects,  the  characteristics  of  the  youths  and  mentors,  and 
information  on  youth-mentor  matching.  The  only  “outcome”  information 
thus  far  provided,  however,  is  information  on  how  satisfied  youths  and 
mentors  were  with  the  mentoring  experience  and  how  much  benefit  each 
perceived  was  derived  from  the  experience.  None  of  the  three  analytic 
approaches  described  above  has  been  successfully  applied  to  study 
outcomes  because  of  a  variety  of  pitfalls  experienced  by  the  national 
evaluation  team,  most  notably  insufficient  data  on  school  performance 
and  behavioral  measures  (e.g.,  delinquent  behavior  and  arrests). 

GAO  assessment:  The  researchers  are  employing  multiple  and 
innovative  strategies  to  determine  the  effectiveness  of  JUMP  in  achieving 
its  objectives.  It  is  not  clear,  however,  whether  definitive  evaluation 
results  can  be  reached  in  the  absence  of  outcome  data  on  youths  who,  in 
the  same  project  areas  at  the  same  points  in  time,  do  not  receive  the 
program.  In  addition,  data  limitations,  if  not  corrected,  may  be  serious 
enough  to  compromise  findings. 


^  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  pointed  out  that 
the  latest  award  sets  project  and  budget  end-dates  of  September  30, 2002.  However,  she 
expects  the  evaluation  will  not  be  completed  before  September  30, 2004.  OJP  does  not 
move  the  project  period  forward  beyond  the  budget  period  as  this  may  imply  an  assurance 
of  future  funding. 
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Partnerships  to  Reduce 
Juvenile  Gun  Violence 
Program 


Program  Description:  The  Partnerships  to  Reduce  Juvenile  Gun 
Violence  Program  is  a  multi-year  demonstration  program  planned  for  four 
sites  (Baton  Rouge  and  Shreveport,  LA;  Syracuse,  NY;  and  Oakland,  CA).  It 
began  in  1997  and  is  expected  to  conclude  in  200L  However,  one  site, 
Shreveport,  was  dropped  from  the  program  early.  The  program  aims  to 
reduce  youth  gun  violence  by  enhancing,  in  specific  target  areas  of  these 
cities,  prevention  and  intervention  strategies  and  strengthening 
partnerships  among  community  residents,  law  enforcement  agencies,  and 
the  juvenile  justice  system.  The  program  involves  mobilizing  the 
community,  establishing  agency  linkages,  and  planning  case  management 
for  juveniles  with  gun  charges  in  year  1,  linking  at-risk  youths  to  services 
in  year  2,  and  expanding  opportunities  for  youths  in  year  3. 

Evaluator:  COSMOS  Corporation. 

Evaluation  description:  The  strategy  for  evaluating  the  impact  of  this 
program  has  evolved  as  the  program  has  unfolded.  An  impact  evaluation 
was  planned  for  three  sites  and  was  to  include  (1)  a  comparison  of 
changes  in  crime  rates  in  target  areas  of  these  cities  before  and  after  the 
implementation  of  the  program,  (2)  a  comparison  of  responses  from  high- 
risk  youths  in  targeted  areas  surveyed  before  and  after  the  program  was 
implemented  and  services  were  provided,  and  (3)  information  on  changes 
in  policies  and  caseloads  revealed  through  focus  group  meetings  and 
interviews  with  agency  officials.  Crime  rate  information  has  thus  far  been 
reported  only  for  Oakland  and  Baton  Rouge,  and  surveys  have  been 
conducted  only  in  Baton  Rouge.  In  Baton  Rouge,  surveys  were  given  to  92 
high-risk  youths  in  the  criminal  justice  system  identified  through  a  variety 
of  processes.  The  sampling  strategies  for  surveying  these  high-risk  youths 
were  unlikely  to  yield  generalizable  results.  In  addition,  fifth-,  seventh-, 
and  ninth-grade  students  in  six  schools  in  the  target  area  were  surveyed  in 
March  of  1999.  It  is  unclear  why  these  students  and  schools  were  sampled 
and  what  response  rates  were.  In  2000,  a  small  sample  of  50  youths  in 
Baton  Rouge  was  identified  as  a  possible  matched  comparison  group  for 
arrest  rate  comparisons.  At  the  time  of  our  review,  this  evaluation  had 
received  $1.2  million  in  funding,  although  a  process  evaluation  is  also 
being  conducted  with  these  funds. 

Evaluation  findings:  The  researchers  report  decreases  in  gun-related 
homicides  and  arrests  in  Oakland  that  were  larger  in  the  target  area  than 
for  the  city  as  a  whole.  They  also  report  decreases  in  gun-related 
homicides  in  Baton  Rouge.  No  analyses  of  results  from  the  survey  data 
have  been  reported  to  date. 
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GAO  assessment:  Comparisons  between  crime  rates  in  the  target 
community  and  the  city  as  a  whole  may  not  be  appropriate.  Student  and 
school  selection  criteria  are  unclear,  making  it  difficult  to  assess  their 
appropriateness  for  obtaining  definitive  results.  In  addition,  if  supporting 
survey  and  administrative  data  are  only  gathered  in  one  site,  it  will  be  very 
difficult  to  generalize  findings  whether  they  appear  positive  or  not. 


Impact  Evaluation  of 
OJJDP  Program — 

Summary  of 
Evaluation  We  Did 

Not  Assess _ _ 

Teen  Courts  The  purpose  of  the  Teen  Courts  evaluation  is  to  measure  the  effect  of 

handling  young,  relatively  nonserious  violators  of  the  law  in  teen  courts, 
rather  than  in  traditional  juvenile  family  courts.  Although  teen  courts  often 
include  many  of  the  same  steps  used  by  formal  juvenile  courts  (for 
example,  intake,  preliminary  review  of  charges,  court  hearing,  and 
sentencing),  they  differ  from  formal  courts  in  that  young  people  are  able  to 
assist  in  the  community  decision-making  process  for  dealing  with  juvenile 
offenders.  Youths  may  act  as  prosecutors,  defense  counsel,  jurors,  court 
clerks,  bailiffs,  and  judge  (or  as  a  panel  of  judges).  To  evaluate  teen  courts, 
both  a  process  and  impact  evaluation  are  used,  with  case  studies  and 
comparison  groups  as  part  of  the  research  design.  In  each  of  the  four  case 
study  sites  (Anchorage,  AK;  Independence,  MO;  Maricopa  County,  AZ;  and 
Rockville,  MD),  data  are  collected  on  about  100  youths  handled  in  teen 
courts  (experimental  group)  and  100  youths  handled  in  the  traditional 
juvenile  justice  system  (comparison  group).  Data  are  also  collected  on 
several  dimensions  of  program  outcomes,  including  post-program  changes 
in  teens’  perceptions  of  justice  and  their  ability  to  make  more  mature 
judgements  as  a  result  of  the  program.  A  process  evaluation  of  the 
projects — exploring  legal,  administrative,  and  case-processing  factors  that 
hinder  the  achieving  of  project  goals — ^is  also  being  conducted. 
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This  appendix  contains  summaries  of  the  24  evaluations  the  Office  of 
Juvenile  Justice  and  Delinquency  Prevention  (OJJDP)  has  fimded  since 
1995  (excluding  the  11  impact  evaluations  it  has  funded  of  its  own 
programs  discussed  in  app.  Vn).  For  22  summaries,  we  used  descriptions 
of  evaluations  that  were  provided  to  us  by  OJJDP;  for  2  summaries,*  we 
wrote  the  descriptions  based  on  OJJDP  documents.  OJJDP  categorized  the 
24  evaluations  into  the  following  three  groups: 

•  OJJDP-fimded  programs:  nonimpact  evaluations  (11). 

•  Non-OJJDP-funded  programs:  impact  evaluations^  (9). 

•  Non-OJJDP-funded  programs:  nonimpact  evaluations  (4). 


OJJDP-Funded 
Programs:  11 
Nonimpact 
Evaluations 

Commimity  Assessment  The  purpose  of  this  evaluation  is  to  test  the  feasibility  and  effectiveness  of 

Centers  OJJDP  community  assessment  center  concept  in  different 

environments.  Community  assessment  centers  seek  to  facilitate  earlier 
and  more  efficient  delivery  of  prevention  and  intervention  services  at  the 
front  end  of  the  juvenile  justice  system.  The  evaluation  uses  a  two-phase 
process  to  (1)  measure  some  outcomes  at  the  two  enhancement  sites,  with 
quasi-experimental  design,®  and  (2)  achieve  more  and  better  outcome 
measures.  But,  according  to  OJJDP,  implementation  and  data  problems 
will  limit  the  effectiveness  of  the  quantitative  methods  employed.  OJJDP 
also  believes  that  the  attempt  to  implement  a  random  assignment  study  at 
one  project  site  will  probably  need  to  be  abandoned.  The  first  phase 
covers  the  four  project  sites  that  comprised  the  Community  Assessment 
Centers  program — ^two  of  these  sites  funded  enhancements  to  existing 


^  We  used  OJJDP  documents  to  develop  summaries  for  SafeFutures  and  Coping  With  Life 
Course. 

®  Impact  evaluations  assess  the  extent  to  which  a  program  causes  changes  in  the  desired 
direction  in  the  target  population. 

®  In  a  quasi-experimental  design,  methods  other  than  random  assignment  are  used  to  create 
a  comparison  group.  When  randomly  assigning  individuals  to  a  control  group  is  not 
feasible,  quasi-experimental  impact  evaluations  can  be  used  to  compare  the  performance 
of  program  participants  on  various  measures  to  individuals  not  in  the  program. 
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programs  and  the  other  two  funded  the  planning  and  implementation  of 
new  programs.  The  second  phase  covers  the  two  project  sites — one 
enhancement  and  one  planning — ^in  which  the  program  is  being  continued 
after  the  end  of  the  first  funding  cycle.^  Many  of  the  evaluation  measures 
are  at  the  project  or  community  level  rather  than  at  the  participant  level. 


Community  Prevention  The  purpose  of  this  evaluation  is  to  examine  the  viability  and  effectiveness 
Grants  Program  (Title  V)  of  the  community-based  delinquency  prevention  model  used  by  grantees  in 

the  Community  Prevention  Grants  Program.  The  Community  Prevention 
Grants  Program  encourages  communities  to  develop  comprehensive, 
collaborative  plans  to  prevent  delinquency.  The  evaluation  focuses  on  two 
main  questions:  (1)  What  is  the  impact  of  the  program  on  community 
planning,  service  delivery,  risk  factors,  protective  factors,  and  juvenile 
problem  behaviors?  (2)  What  factors  and  activities  lead  to  the  effective 
implementation  of  the  Community  Prevention  Grants  Program  model  and 
to  positive  program  outcomes?  This  evaluation  employs  a  case  study 
approach  supplemented  by  a  basic  profile  of  communities  that  are 
participating  in  the  program.  Case  studies  are  to  be  implemented  in  11 
communities  in  6  states.  Evaluation  measures  are  to  be  applied  at  the 
project,  community,  and  program  levels. 


Comprehensive  Strategy 
for  Serious,  Violent,  and 
Chronic  Juvenile 
Offenders  (Comprehensive 
Strategy) 


The  purpose  of  this  process  evaluation^  is  to  address  the  following 
questions  about  the  Comprehensive  Strategy  program:  (1)  What  are  the 
factors  associated  with  successful  Comprehensive  Strategy  planning  and 
implementation?  (2)  To  what  extent  do  project  sites  adhere  to  the 
prescribed  Comprehensive  Strategy  framework?  (3)  What  are  the  mqjor 
implementation  challenges  program  grantees  face  in  implementing  the 
Comprehensive  Strategy?  (4)  To  what  extent  does  the  training  and 
technical  assistance  provided  to  project  sites  help  them  acquire  the 
knowledge,  skill,  and  tools  necessary  to  develop  the  Comprehensive 
Strategy?  (5)  What  role  should  OJJDP  play  in  the  future  implementation  of 
the  Comprehensive  Strategy?  The  Comprehensive  Strategy  is  OJJDP’s 


^  In  commenting  on  a  draft  of  this  report,  the  Assistant  Attorney  General  explained  that  the 
other  two  project  sites  were  not  included  in  the  second  phase  of  the  evaluation  because 
they  did  not  receive  additional  funding.  One  site  is  no  longer  a  separate  project  and  the 
other  project — ^a  planning  site — continues  to  move  toward  operational  status. 

^  A  process  evaluation  uses  empirical  data  to  assess  delivery  of  a  program  and  verifies 
whether  it  was  delivered  as  intended  to  the  targeted  recipients. 
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approach  for  addressing  juvenile  violence  and  delinquency  at  the 
community,  state,  and  national  levels  through  a  systematic  plan.  It 
advocates  the  use  of  local  planning  teams  to  assess  the  factors  and 
influences  that  put  youths  at  risk  of  delinquency,  determine  available 
resources,  and  establish  prevention  programs  to  either  reduce  risk  factors 
or  provide  protective  factors  that  buffer  juveniles  from  the  impact  of  risk 
factors.  This  evaluation  uses  a  multilevel  design  to  assess  how  project 
sites  implement  the  Comprehensive  Strategy.  The  evaluation  began  with 
telephone  interviews  with  site  coordinators  from  all  48  project  sites;  25  of 
the  48  project  sites  were  randomly  selected  for  a  stakeholder  survey.  One 
year  later,  10  of  the  25  project  sites  are  being  given  a  second  stakeholder 
survey.  Subsequently,  five  sites  are  to  be  selected  for  visits,  and  intensive 
case  studies  are  being  done  in  three  cities. 


Drug-Free  Communities 
Support  Program 


The  purpose  of  this  evaluation  is  to  examine  (1)  community  coalitions’ 
developmental  processes  from  the  early  planning  and  adoption  stages 
through  implementation  and  later  stages  and  (2)  the  impact  of  coalitions’ 
prevention  efforts  concerning  risk  and  resiliency  factors  and,  to  the  extent 
feasible,  alcohol,  tobacco,  and  other  drug  use.  The  Drug-Free 
Communities  Support  Program  provides  grants  to  community  coalitions  to 
strengthen  their  efforts  to  prevent  and  reduce  yoimg  people’s  filegal  use  of 
drugs,  alcohol,  and  tobacco.  The  evaluation  is  studying  two  cohorts  of 
program  grantees — ^those  that  received  grants  in  1998  or  1999  (cohort  1) 
and  those  that  received  grants  in  2000  (cohort  2).  The  national  evaluation 
sample  is  comprised  of  a  total  of  213  grantees.  The  sample  is  divided  by 
years  of  operation:  1-5  years,  6-9  years,  and  more  than  9  years. 
Semiannually,  cohort  1  grantees  are  required  to  submit  progress  reports  to 
OJJDP  and  the  evaluator  that  include  a  special  section  (Part  II),  which 
provides  information  about  the  compositions  of  the  coalitions  and 
outcome  data  collection.  Cohort  2  grantees  do  not  have  a  Part  n  reporting 
requirement  and  submit  progress  reports  semiannually  only  to  OJJDP.  In 
addition,  21  grantees  (15  from  cohort  1  and  6  from  cohort  2)  serve  as 
intensive  study  sites,  where  interviews  with  staff  and  stakeholders  provide 
greater  detail  about  coalition  development  and  local  program  evaluation. 


Juvenile  Accountability 
Incentive  Block  Grants 
Program 


The  purpose  of  this  process  evaluation  is  to  provide  feedback  to  OJJDP  on 
the  implementation  of  the  Juvenile  Accountability  Incentive  Block  Grants 
Program.  The  program  encourages  states  and  local  jurisdictions  to 
implement  accountability-based  programs  and  services  in  54  states  and 
U.S.  territories.  The  evaluator  is  surveying  state  and  local  practitioners, 
policy  makers,  and  grant  program  administrators  about  their  perceptions 
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and  attitudes  about  the  program  and  its  administration.  Specifically,  the 
evaluation  focuses  on  (1)  understanding  how  states  and  local  umts  of 
government  plan  for  and  administer  program  funds  and  (2)  examining  the 
perceptions  of  states  and  local  units  of  government  about  how  weU  the 
program  is  achieving  congressional  intent.  In  addition,  in-depth  case 
studies  are  conducted  at  a  limited  number  of  sites. 


Performance-based  The  purpose  of  this  evaluation  is  to  (1)  provide  feedback  to  the 

Standards  Project  Performance-based  Standards  Project  team  on  improving  design  and 

implementation  support  to  the  sites,  (2)  assist  the  project  team  in  refining 
the  Performance-based  Standards  Project  model  and  in  maximizing 
responsiveness  to  the  needs  of  the  participants,  that  is,  those  who  are 
implementing  the  project  model,  and  (3)  chronicle  the  development  of  the 
project  and  summarize  lessons  learned.  OJJDP  established  the 
Performance-based  Standards  Project  to  improve  the  quality  and 
conditions  of  juvenile  corrections  facilities.  Specifically,  the  project 
develops  and  implements  outcome  standards  and  an  assessment  tool. 
Corrections  facilities  can  use  both  to  monitor  progress  towards  meetings 
goals  in  areas  of  operations,  such  as  health  and  safety.  The  evaluation  uses 
a  case  study  approach.  This  approach  consists  of  the  collection  of  both 
quantitative  and  qualitative  data  describing  the  processes  used  to 
implement  the  project  model  in  80  juvenile  detention  and  correctional 
facilities  across  the  country.  Site  visits  are  made  and  in-depth  case  studies 
are  planned.  An  all-site  survey  is  distributed  to  key  participants  to 
determine  satisfaction  with  the  supports  provided  to  them  in  the 
implementation  of  the  project  model.  In  addition,  the  survey  seeks  the 
participants’  assessment  of  (1)  the  impact  the  project  has  made  on 
conditions  of  confinement  and  management  of  the  facilities  and  (2)  the 
overall  utility  of  the  project  model. 


Prenatal  and  Early 
Childhood  Nurse  Home 
Visitation  Program 


The  purpose  of  this  evaluation  is  to  (1)  determine  the  extent  to  which 
replication  project  sites  have  been  able  to  conform  to  the  original  program 
model,  and  (2)  assess  the  “prosocial”  (that  is,  positive,  socially-oriented 
behavior)  outcomes  for  mothers  and  tiieir  babies.  The  Prenatal  and  Early 
Childhood  Nurse  Home  Visitation  Program  consists  of  intensive  and 
comprehensive  home  visitation  by  nurses  during  a  woman’s  pregnancy 
and  the  first  2  years  following  the  birth  of  her  first  child.  The  evaluation 
involves  six  project  sites  and  employs  a  quasi-experimental  design  with 
matched  comparison  groups. 
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Partnerships  to  Reduce 
Youth  Violence  and 
Delinquency  Program 
(SafeFutures) 


This  purpose  of  this  process  evaluation  is  to  document  and  understand  the 
process  of  community  mobilization  and  collaboration.  SafeFutures  is 
designed  to  build  a  comprehensive  program  of  prevention  and  intervention 
strategies  for  at-risk  youths  and  juvenile  offenders.  The  program 
comprises  six  project  sites  that  represent  urban,  rural,  and  Native 
American  communities.  The  evaluation  is  examining  all  six  project  sites. 
Project  sites  collect  and  record  performance  data  on  program  operations 
and  client  outcomes  using  the  Client  Indicator  Data  Base.  The  project  sites 
are  required  to  collect  extensive  information  from  selected  SafeFutures 
program  components  on  individual  participants’  risk  and  protective  factor 
profiles,  youths’  service  utilization,  and  agencies’  coordination  of  services 
for  youth  during  the  course  of  their  involvement  in  the  SafeFutures 
program.  In  addition,  they  collect  information  on  outcome  measures 
regarding  youths’  educational  commitment  (that  is,  school  attendance, 
achievement,  and  behavior),  youths’  involvement  in  delinquency  and 
crime,  and  any  changes  in  youths’  risk  profiles.  Analysis  of  these  data  will 
provide  a  picture  on  program  performance  in  three  key  areas:  reaching  the 
intended  high-risk  youth  clientele,  coordinating  services  for  youths  with 
multiple  problems,  and  monitoring  subsequent  school  performance 
problems  and  involvement  in  the  juvenile  justice  system. 


Safe  Kids/Safe  Streets  The  purpose  of  this  evaluation  is  to  document  the  lessons  learned  and 

factors  associated  with  the  successful  development  and  implementation  of 
the  Safe  Kids/Safe  Streets  program.  The  Safe  Kids/Safe  Streets  program  is 
designed  to  (1)  help  communities  break  the  cycle  of  early  childhood 
victimization  and  later  criminality  and  (2)  reduce  child  abuse  and  neglect, 
as  well  as  the  child  fatalities  that  often  result.  The  evaluation  is  surveying 
five  project  sites  through  five  data  collection  strategies:  agency 
administrative  data,  case  tracking,  key  informant  interviews,  surveys  of 
agency  professionals,  and  surveys  of  stakeholders. 


Tribal  Youth  Program  The  purpose  of  this  evaluation  is  to  (1)  support  culturally  appropriate 

process  and  outcome  evaluations  of  activities  funded  under  Tribal  Youth 
Program  grants  and  (2)  build  the  capacity  of  tribes  to  better  evaluate  their 
own  juvenile  justice  programs  and  activities.  The  Tribal  Youth  Program 
assists  grantees  in  developing  projects,  within  tribal  communities,  for  the 
prevention  and  control  of  youth  violence  and  substance  abuse.  The 
evaluation  is  participatory  in  nature,  that  is,  project  personnel  and 
stakeholders  will  be  involved  in  developing  the  evaluation  designs,  with 
the  assistance  and  guidance  of  an  evaluation  facilitator.  The  five  project 
sites  are  implementing  different  projects  and  have  not  yet  completed  their 
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evaluation  designs.  According  to  OJJDP,  it  is  too  early  in  the  evaluation  to 
tell  exactly  what  designs  are  to  be  used.  OJJDP  has  required  that  aU 
evaluations  be  designed  to  examine  both  program  implementation  and 
program  outcomes. 


Truancy  Reduction  The  pmpose  of  this  process  evaluation®  is  to  (1)  determine  how 

Pro2ram  commumty  collaboration  can  affect  truancy  reduction  and  lead  to 

systemic  reform  and  (2)  assist  OJJDP  in  the  development  of  a  model  for  a 
truancy  reduction  program,  including  identifying  the  essential  elements  of 
that  model.  The  Truancy  Reduction  Demonstration  Program  encourages 
communities  to  develop  comprehensive  approaches — involving  schools, 
parents,  the  justice  system,  law  enforcement,  and  social  service 
agencies — in  identifying  and  tracking  truant  youths.  The  evaluation  is 
employing  site  visits,  interviews  with  key  personnel,  and  case  studies  of 
individual  sites.  Process  data  are  gathered  from  all  seven  project  sites 
participating  in  the  evaluation  and,  from  some  sites,  limited  outcome  data 
are  gathered. 


N  on-0 JJDP-Funded 
Programs:  Nine 
Impact  Evaluations 


Adolescent  Female 
Offenders  (Three 
Programs) 


The  purpose  of  this  evaluation  is  to  evaluate  the  efficacy  of  three 
Adolescent  Female  Offenders  programs  in  Wayne  County,  Michigan,  The 
three  programs  are  (1)  a  program  incorporating  gender-specific 
programming,  home-based  intervention,  and  community  involvement, 
including  pregnant  and  parenting  adolescents;  (2)  an  intensive  probation 
program  with  limited  gender-specific  programming;  and  (3)  a  traditional, 
female-only  residential  program  that  provides  limited  gender-specific 
training.  The  evaluation  is  using  a  quasi-experimental  design.  Using 
random  assignment,  the  home-based  intervention  model  is  to  be  compared 
with  the  established  intensive  probation  model;  the  outcomes  of  these 
models  are  then  to  be  compared  with  outcomes  of  the  traditional. 


^  The  Truancy  Reduction  Demonstration  Program  and  its  evaluation  are  jointly  funded  by 
OJJDP,  the  Office  of  Justice  Program’s  Executive  Office  of  Weed  and  Seed,  and  the  U.S. 
Department  of  Education. 
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female-only  residential  program.  The  comparison  analysis  involves  at  least 
50  young  women  in  each  of  the  3  programs.  A  wide  range  of  outcomes — 
including  recidivism,  substance  use,  depression,  community  integration, 
academic  performance  and  career  aspirations,  parenting  readiness,  and 
responsible  sexual  behavior — ^is  to  be  examined.  The  evaluator  is  also 
exploring  the  relationship  of  specific  program  components  to  these 
outcomes. 


Coping  with  Life  Course  The  purpose  of  this  impact  evaluation  is  to  evaluate  the  effectiveness  of  a 

cognitive-behavioral  group  intervention.  The  Coping  With  Life  Course  is 
aimed  at  enhancing  prosocial  coping  and  problem  solving  for  adolescents 
incarcerated  in  youth  correctional  facilities.  To  evaluate  the  program,  a 
minimum  of  120  adolescents  in  one  youth  correctional  facility  are 
randomly  assigned  to  either  the  Coping  with  life  Course  intervention 
group  or  a  standard-care  control  group.  Six  Coping  with  life  Course 
cohort  groups  of  10  each  are  followed.  The  evaluation  is  allowing  for 
attrition  (firom  ttie  initial  60  participants  down  to  48)  in  each  of  the 
intervention  and  control  groups.  Participant  functioning  is  assessed  before 
and  after  intervention  through  a  battery  of  questiormaires.  Recidivism, 
return  to  close  custody,  and  service  utilization  are  tracked  through 
databases  and  statewide  records. 


Creation  and 
Implementation  of  a 
Family  Index  in  Riverside 
County,  California,  Courts 


The  piupose  of  the  evaluation  is  to  (1)  document  the  implementation  of  a 
new  “family  index”  case  management  system  (through  a  process 
evaluation)  and  (2)  examine  the  impact  of  the  family  index  on  juvenile 
court  case  processing  (through  an  impact  evaluation).  The  family  index 
system  allows  cross-referencing  to  identify  all  family  members  involved  in 
family  law;  juvenile  dependency;  juvenile  delinquency;  and  criminal,  civil, 
and  probate  matters.  For  the  process  evaluation,  a  case  study  approach  is 
used  to  describe  the  implementation  of  the  family  index  at  one  project 
site,  the  Riverside,  California,  Court.  For  the  impact  evaluation,  a  pre-post 
design  is  used  to  examine  how  the  family  index  has  affected  juvenile  court 
matters  (for  example,  court  processing  time,  coordination  between  courts, 
and  content  of  hearings). 


Flashpoint  The  purpose  of  this  evaluation  is  to  evaluate  the  effects  of  Flashpoint  on 

the  antisocial  patterns  of  juvenile  offenders’  thoughts  and  actions  and  high 
school  students’  thoughts  and  actions.  Specifically,  changes  are  assessed 
for  (1)  media  use  and  literacy,  (2)  violence-supporting  beliefs  and 
behavior,  and  (3)  substance  use  and  abuse.  The  Flashpoint  program  is 
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designed  to  build  critical  thinking  skills  young  people  need  to  (1)  see 
through  false  media  portrayals  that  glorify  violence  and  drug  use  and  (2) 
apply  decision-making  in  their  own  lives.  The  evaluation  is  using 
comparison  groups,  pre-post  interventions,  and  case  studies.  Participants 
include  264 juveniles,  ages  14  to  17.  Treatment  groups  are  compared  with 
no-treatment  control  groups  for  baseline-to-posttest  changes.  Three 
groups  and  project  sites  are  involved:  (1)  repeat  and  serious  offenders  in  a 
correctional  program,  (2)  first-time  offenders  in  a  diversion  program,  and 
(3)  students  in  a  public  high  school. 


Free  to  Grow:  Head  Start 
Partnerships  to  Promote 
Substance-Free 
Communities 


The  purpose  of  this  evaluation  is  to  determine  if  Free  to  Grow  can  reduce 
substance  abuse  (alcohol  use,  smoking,  and  illegal  drug  use).  The  Free  to 
Grow  program  builds  on  existing  Head  Start  programs,  adding  community- 
strengthening  and  family-intervention  components  to  address  the  problem 
of  substance  abuse.  The  evaluation  is  attempting  to  determine  the 
independent  effects  of  these  two  components  on  substance  abuse 
prevention.  It  involves  16  project  sites  and  16  comparison  sites  and 
employs  a  multistage  experimental  research  design. 


Gaining  Insight  Into 
Relationships  for  Lifelong 
Success  Project 


The  purpose  of  this  evaluation  is  to  provide  a  process  and  outcome 
evaluation  of  the  Gaining  Insight  into  Relationships  for  Lifelong  Success 
Project.  The  project  involves  two  primary  levels  of  intervention:  (1)  a 
psycho-educational  counseling  group,  dealing  with  relationships  and 
involving  girls  in  four  relational  domains  (relation  to  self,  family,  peers, 
and  teachers),  and  (2)  a  focus  on  individual  consultations,  educational 
workshops  and  the  policies  and  procedures  of  the  local  juvenile  justice 
system,  and  the  involvement  of  court  service  workers  from  the  system. 
Specifically,  the  evaluation  will  (1)  investigate  the  applicability  of  a 
relational  approach  to  the  treatment  of  female  juvenile  offenders;  (2) 
examine  the  components  of  the  relational  j^proach  that  deal  with 
relationships  to  self,  family,  peers  and  teachers;  (3)  evaluate  the  impact  of 
increasing  the  knowledgebase  of  professionals  involved  in  the  local 
juvenile  justice  system;  and  (4)  provide  an  empirically  based,  alternative 
treatment  model  that  can  be  replicated  in  other  settings.  The  evaluation  of 
the  first  level  of  intervention — ^the  counseling  group — ^focuses  on  each  of 
the  four  relational  domains  through  the  use  of  multimethod  data 
collection;  this  collection  includes  self-reports  and  other  reports,  school 
records,  and  recidivism  data.  The  evaluation  of  the  second  level  of 
intervention  focuses  on  the  court  services  that  workers  use,  specifically 
gender-sensitive  treatment  recommendations  and  referrals;  qualitative 
observational  data,  gathered  from  monthly  meetings,  will  be  used.  There  is 
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random  assignment  between  girls  referred  to  either  the  project 
intervention  or  to  the  standard  intervention  currently  being  used  by  the 
Clark  Coimty  Court  in  Athens,  Georgia.  Approximately  180  girls — ^90 
referred  to  the  project  intervention  and  90  referred  to  the  standard  court 
intervention — are  to  be  evaluated. 

Quantum  Opportunities 

The  purpose  of  this  evaluation  is  to  assess  the  implementation  and  the 
impact  of  Quantum  Opportunities.  The  Quantum  Opportunities  program  is 
designed  to  reduce  the  incidence  of  delinquency,  criminal  behavior,  and 
subsequent  involvement  in  the  criminal  and  juvenile  justice  systems 
amongst  educationally  at-rlsk  inner  city  youths.  The  evaluation  is  using  an 
experimental  design  with  random  assignment.  Ninth-grade  students  at  six 
sites  are  randomly  assigned  to  treatment  and  control  groups,  with  the 
treatment  group  enrolled  in  the  Quantum  Opportunities  program.  The 
students  are  followed  through  their  high  school  careers  and  2  years 
beyond.  Information  is  collected  from  academic  achievement  tests, 
administered  each  year,  and  from  two  questionnaires. 

Restorative  Justice 
Conferences 

The  purpose  of  this  evaluation  is  to  test  the  impact — on  recidivism, 
program  completion,  and  victim  satisfaction— of  the  Restorative  Justice 
Conferences  for  a  population  of  youthful  offenders  (aged  14  and  under)  in 
an  urban  setting  (Indianapolis,  IN).  Restorative  Justice  Conferences  bring 
together  the  offender,  victim,  and  supporters  of  each  so  as  to  provide  an 
opportunity  for  fuller  discussion  of  the  offense;  the  effect  of  the  offense  on 
the  victim,  the  offender’s  family,  and  greater  coitununity;  and  steps  the 
offender  can  take  to  make  amends.  The  evaluation  is  using  a  single-site 
evaluation  with  an  experimental  design.  As  part  of  the  design,  youths  are 
randomly  assigned  to  a  treatment  group  (Restorative  Justice  Conferences) 
or  a  matched  control  group. 

Risk-Focused  Community 
Policiug 

The  purpose  of  this  evaluation  is  to  determine  if  the  program  reduces  the 
amount  of  delinquency  in  a  city.  The  Risk-Focused  Community  Policing 
program  increases  protection  by  the  community  police,  potentially 
reducing  delinquency.  The  evaluation  is  using  an  experimental  research 
design.  The  project  site  (city)  is  divided  into  approximately  40  census 
blocks,  with  20  blocks  randomly  selected  as  program  blocks  and  the  other 
20  designated  as  control  blocks. 

Page  77 


GAO-02-23  O JJDP  Grantee  Reporting  Requirements  and  Conce 


Appendix  VIII:  Twenty-Four 

Other  Evaluations  Funded  by 

OJJDP 

Non-OJJDP-Funded 

Programs:  Four 

Nonimpact 

Evaluations 

Act  Now  Truancy  Program 

The  purpose  of  this  evaluation  is  to  study  the  Act  Now  Truancy  Program. 
The  program  is  a  prosecutor-led  truancy  reduction  program.  The 
evaluation  is  using  a  pre-post  intervention  design  involving  one  project 
site.  Information  is  collected  and  aggregated  (for  example,  truancy  rates 
rather  than  individual  truancy  behavior)  for  all  participants. 

Childhood  Violence 
Prevention  Program  (Let’s 
Talk  About...) 

The  purpose  of  this  evaluation,  conducted  in  two  schools,  is  to  assess  the 
impact  of  the  Childhood  Violence  Prevention  Program.  The  program  is 
designed  to  prevent  the  legitimization  of  aggression  among  pre-adolescent, 
elementary,  and  middle  school  children,  with  special  focus  on  victims  of 
child  maltreatment.  The  evaluation  is  using  a  pre-post  intervention  design, 
with  comparison  groups.  The  study  involves  having  elementary  school 
students  participate  in  a  class  activity  using  a  workbook  designed  to 
encourage  problem  solving  action  rather  than  aggressive  behavior  in 
interactions  with  peers. 

Community-Level 

Programs  for  Youth 

This  project  is  not  an  evaluation,  per  se,  but  rather  a  synthesis  of  existing 
evidence  on  community-level  interventions  and  service  programs.  Its 
purpose  is  to  identify  the  strengths  and  weaknesses  of  communily-level 
evaluations  and  to  provide  recommendations  to  the  field  about  how  to 
structure  and  carry  out  such  evaluations.  Community-level  programs  for 
youths  are  designed  to  promote  positive  youth  development.  To  evaluate 
the  programs,  a  committee — experts  from  several  disciplines  (child  and 
adolescent  development,  child  health,  sociology,  psychology,  evaluation 
research,  youth  services,  and  community  development) — ^is  assessing  the 
strengths  and  limitations  of  measurements  and  methodologies  that  have 
been  used  to  evaluate  these  interventions. 
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Department  of  Labor’s 
Education  and  Training  for 
Youthful  Offenders 
Initiative 


The  purpose  of  this  evaluation  is  to  assess  the  activities  undertaken  by 
project  sites,  determine  whether  they  can  be  evaluated,  and  ultimately 
assess  the  impact  of  these  activities  on  the  youthful  offenders  participating 
in  the  program.  The  program  is  intended  to  (1)  enhance  school-to-work 
education  and  training  in  juvenile  correctional  facilities  and  (2)  improve 
youthful  offenders’  transition  into  the  community.  The  evaluation  design 
has  not  been  completed,  but  random  assignment  study  is  strongly 
preferred,  if  feasible.  At  the  time  of  our  review,  only  one  of  the  three 
potential  sites  could  be  evaluated.  One  more  project  site  is  to  be  awarded 
and,  if  it  can  be  evaluated,  it  will  be  added  as  a  second  evaluation  site. 


Page  79 


GAO-02-23  OJJDP  Grantee  Reporting  Requirements  and  Conce 


Appendix  IX:  Comments  from  the 
Department  of  Justice 


Note:  GAO  comments 
supplementing  those  in 
the  report  text  appear  at 
the  end  of  this  appendix. 


U.S.  Department  of  Justice 
Office  of  Justice  Programs 


Office  of  the  Assistant  Attorney  General  Washington,  DX.  20531 


OCT  1  5  20D1 

Laurie  E.  Ekstrand 
Director 
Justice  Issues 

Tax  Administration  and  Justice 
General  Accounting  Office 
441  G  Street,  N.W.  Room  2A38 
Washington,  D.C.  20548 

Dear  Ms.  Ekstrand: 

This  correspondence  responds  to  tiie  General  Accounting  Office  (GAO)  draft  report  entitled 
** JUVENILE  JUSTICE:  Information  on  OJJDP  Grantee  Reporting  Requirements  and  Concerns 
About  Evaluation  Studies*’  (GAO-02-23).  The  draft  report  has  been  reviewed  by  the  Office  of 
Justice  Programs’  (OJP’s)  Office  of  Juvenile  Justice  and  Delinquency  Prevention’s  (OJJDP’s) 
Research  and  Program  Development,  Training  and  Technical  Assistance,  and  Special  Emphasis 
Divisions. 

Overall,  the  draft  report  provides  useful  information  that  highlights  areas  warranting  attention. 
The  report  will  be  an  important  tool  which  we  will  use  to  improve  the  quality  of  our  evaluations, 
and  ultimately  to  design  programs  that  will  achieve  greater  impact.  The  OJJDP’s  ^search  and 
Program  Development  Division  (RPDD),  in  conjunction  with  the  other  OJJDP  divisions,  will 
commence  the  development  of  a  guide  and  related  tools  that  communities  can  use  without 
additional  funding  to  assess  both  program  impact,  and  process  activities  related,  to  existing 
services  and  subgrantee  efforts. 

As  recommended,  we  will  assess  the  five  impact  evaluations  that  are  cuirently  in  their  formative 
stage  to  address  potential  comparison  group  and  data  collection  problems.  On  the  basis  of  that 
assessment,  OJJDP  will  initiate  any  needed  interventions  to  help  ensure  that  the  evaluations 
produce  definitive  results. 

This  review  provides  an  important  opportunity  to  bring  to  light  the  difficulties  associated  with 
evaluation  of  real  life  community  bas^  programs  where  the  luxury  of  laboratory-like  comparison 
groups  may  not  be  possible.  We  agree  with  GAO  that  we  should  always  strive  for  more  rigorous 
and  scientifically  sound  evaluation  designs.  However,  we  were  disappoint^  in  GAO’s  reliance 
on  the  use  of  a  comparison  groiq>  as  the  only  valid  evaluation  design.  The  inclusion  of 
comparison  groups  would  certainly  strengthen  the  int^pretation  of  evaluation  results.  However, 
RPDD,  does  not  have  sufficient  funding  to  include  comparison  groups  in  every  evaluation. 
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Now  on  pp.  3,  8-10,  and 
22-25. 

Now  on  pp.  23-24. 


Given  the  choice  between  conducting  fur  fewer  evaluations,  all  with  companson  groups,  and 
conducting  a  greater  number  of  evaluations  under  less-than-ideal  conditions,  the  RPDD  worics 
hard  to  tread  a  middle  groimd  which  satisfies  needs  for  both  quality  and  quantity. 

The  GAO  discusses  evaluation  methodology  throughout  the  report  (pp.  3, 7,  8, 21, 23, 25,  and 
26)  and  the  “lack  of  appropriate  comparison  groups.”  For  example,  on  page  23  of  the  repor^ 
GAO  provides  a  hypothetical  example  of  what  it  believes  is  an  appropriate  use  of  a  comparison 
group  in  a  program  designed  to  improve  students’  grades.  We  note  that  an  education  example  in 
a  controlled  classroom  environment  was  profiled,  not  a  juvenile  justice  example  in  a  commumly 
setting,  which  would  have  been  more  relevant.  Regarding  a  hypothetical  comparison  group  GAO 
states,  ” a  second  set  of  students  who  are  not  in  the  program  but  are  matched  in  academic 
performance  and  exposed  to  all  of  the  same  factors  (except  the  propm)  could  provide  a  baseline 
from  which  to  assess  the  impact  of  the  program."  We  agree  with  this  statement  in  theory,  but 
outside  highly  controlled  laboratory  settings,  there  is  no  way  to  assure  that  participants  are 
exposed  to  "all  of  the  same  factors  (except  the  program)."  The  best  that  can  be  done  in  a  real 
world  setting  is  to  randomly  assign  similar  participants  to  treatment  and  comparison  groups  and 
hope  that  any  other  effects  distribute  themselves  randomly  and  wash  out  (i.e.,  become  random 
error)  in  tiie  analysis. 

OJJDP  seeks  to  conduct  juvenile  justice  evaluations  in  a  real  world  setting.  School  research  has 
become  increasingly  difficult  due  to  the  need  for  obtaining  parental  consent  and  the 
unwillingness  of  schools  to  use  class  time  for  research  programs. 

A  growing  number  of  policy  makers  and  evaluators  firmly  believe  that  commumty  based 
initiatives  do  not  lend  themselves  to  the  kind  of  tradition^  evaluations  that  this  draft  report 
proposes.  In  particular,  the  Annie  E.  Casey  Foundation  has  been  grappling  with  this  issue  for 
years.  Concern  with  the  inadequacies  of  traditional  evaluations  has  given  rise  to  the  Roundtable 
on  Comprehensive  Community  Initiatives  for  Children  and  Family,  a  policy  program  of  the 
Aspen  Institute  which  had  its  origins  with  the  National  Academy  of  Sciences.  Comprised  of  30 
members  who  have  been  engaged  in  cross-system,  geogr^hically  targeted  initiatives,  the 
Roundtable  issued  its  first  report  in  1995  (Connell,  Kubisch  et  al.  1995)  that  strongly  urged  that 
new  approaches  to  evaluation  be  develop^.  This  view  is  based  on  the  feet  that  so  many 
individual  interventions  are  necessary,  but  not  sufficient  to  improve  outcomes. 

We  are  heartened  that  GAO  has,  through  this  effort,  highlighted  the  difficulties  associated  wth 
such  efforts  and  look  forward  to  any  suggestions  the  reviewers  may  tove  in  dealing  wife  these 
real  world  issues.  Our  detailed  comments  are  attached  for  your  consideration. 
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If  you  have  any  qi^stions  or  need  additional  infonnatlon  about  this  response,  please  contact  me 
on  (202)  616-3433  or  LeToya  Bryant,  OJP  Audit  Liaison,  on  (202)  514-0692.  Thank  you  for 
your  continued  cooperation  and  assistance. 

Sincerely, 

Deborah  J.  Daniels 
Assistant  Attorney  General 


Attachment 

cc:  Terrence  S.  Donahue 

Actii^  Administrator 
OfSce  of  Juvenile  Justice  and 
Delinquency  Prevention 

Cynthia  J.  Schwimer 
Comptroller 

Office  of  Justice  Programs 

LeToya  A.  Bryant 

Audit  Liaison 

Office  of  Justice  Programs 

OAAG  Executive  Secretariat 
Control  No.  20012013 
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Attachment 

The  Office  of  Justice  Programs  (OJP)  Specific  Comments  on  the  General  Accounting 

Office  (GAO)  draft  report  entitled:  “JUVENILE  JUSTICE:  Information  on  OJJDP  Grantee 

Reporting  Requirements  and  Concerns  About  Evaluation  Studies**  (040-02-23) 

Now  on  p.  21 . 

1. 

On  p^e  2,  the  report  responds  to  Congressman  Bob  Schaffer’s  question  about  the 
“number  of  juveniles  served",  by  noting  that  "many  of  its  [OJXDP’s]  programs  are  not 
intended  to  serve  juveniles  directly."  Later,  on  pages  19-20,  the  report  elaborates  on  \vhy 
grantees  might  not  be  required  to  report  the  number  of  juveniles  served.  Although 
research  is  mentioned  in  passing,  there  is  no  explanation  of  why  "juvemles  served"  might 
not  be  an  appropriate  measure  for  a  variety  of  research  projects.  There  is  no  mention,  for 
example,  of  the  fact  that  although  research  projects  do  not  typically  provide  services 
directly,  their  results  can  potentially  help  thousands  of  juveniles.  The  report  should  more 
clearly  explain  the  nature  and  importance  of  the  work  OJJDP  does. 

Now  on  p.  4. 

2. 

On  page  3,  “Background”  section,  OJJDP’s  mission  should  be  more  clearly  reflected  to 
include  its  emphasis  on  improving  the  juvenile  justice  system.  It  is  critical  to  reflect  the 
full  mission  statement  in  order  to  provide  clarity  and  context  when  describing  the  types  of 
programs  funded  and  the  extent  to  which  they  serve  juveniles  directly. 

Now  on  p.  6. 

3. 

On  page  5  of  the  draft  report,  “For  example,  under  the  Formula  Grants  Program,  states 
pass  through  two-thirds  of  their  awarded  funds  to  public  and  private  non-profit 
organizations.”  The  words,  “a  minimum  of’  should  be  inserted  before  the  ‘two-thirds’*  as 
there  are  states  that  actually  pass-through  more  than  the  minimum  2/3.  The  statute  reads 
that  at  least  66  and  2/3  must  be  passed  through. 

Now  on  p.  9. 

4. 

On  page  8,  the  report  describes  the  methodology  of  data  collection  for  the  report. 

Specifically,  the  report  states  that  GAO  utilized  a  data  collection  instrument  to 
systematicily  collect  information  on  OJJDP  evaluations  from  the  grant  files.  This 
description  raises  a  few  questions.  First,  how  was  missing  data  treated?  The  GAO 
reviewers  spent  a  great  deal  of  time  here  and  RPDD  staff  coordinated  the  document 
retrieval  for  some  of  the  missing  documents  within  RPDD.  However,  it  is  likely  the  case 
that  there  was  missing  data  because  documents  were  missii^  or  documents  were  not 
complete.  A  fair  analysis  of  the  state  of  evaluation  would  have  to  include  a  strategy  to 
solicit  this  information  for  all  potential  sources.  Had  the  GAO  reviewers  consulted  with 
the  program  managers  directly,  there  would  have  been  an  opportunity  for  OJJDP  program 
managers  to  correct  and  complete  some  of  the  data  that  had  already  been  collected.  The 

GAO  reviewers  should  clearly  present  in  the  draft  report  the  strategy  for  the  treatment  of 
missing  data. 

Now  on  p.  1 0. 

5. 

On  page  9,  (entire  paragraph),  the  report  implies  diat  OJJDP  believes  that  the  semiannual 
progress  report  provides  sufficient  information  vsdth  which  to  monitor  grantee  s  project 
implementation  and  achievement  of  the  goals  they  identified  in  their  grant  applications. 
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Now  on  pp.  10-18. 


Now  on  p.  22. 
See  comment  1 . 


While  OJJDP  has  indicated  that  the  semi-annual  progress  report  is  one  monitoring  tool 
that  is  used  to  meet  basic  oversight  needs,  we  have  also  indicated  that  we  would  prefer, 
and  perhaps  make  better  use  of,  more  frequent  reports  with  more  detailed  information. 

The  GAO  report  did  not  provide  any  recommendations  or  conclusions  directly  related  to 
grantee  reporting  requirements.  For  policy  consideration  purposes,  perhaps  it  would  be 
useful  for  GAO  to  comment  conclusively  on  grantee  reporting  requirements,  instructions, 
and  on  what  is  gained  through  these  reports. 

6.  On  pages  9-21  (entire  section  pertaining  to  grantee  reporting  reqmremcnts),  as  presented, 
it  is  not  readily  apparent  that  this  section  of  friis  report  is  not  limited  to  only  "evaluation” 
grantees.  There  is  considerable  discussion  regarding  Juvenile  Accountability  hw^tive 
Block  Grant  (JAIBO)  reporting  and  the  Formula  report,  and  it  is  clear  that  GAO  is 
making  statements  regarding  the  OJJDP  grantees  reporting  the  number  of  juveniles 
served  that  include  ATT.,  grantees,  not  just  evaluation  grantees.  The  point  is  important 
because,  at  first  look,  it  spears  that  the  report  suggests  (as  does  the  title)  that  the 
reporting  issues  raised  are  limited  to  evaluation  studies.  We  propose  that  the  report’s  title 
be  modified  to  “Juvenile  Justice:  Information  on  OJJDP  Grantee  Reporting  Requirements 
for  Discretionary  and  Formula  Grant  Programs  and  Concerns  About  Discretionary 
Evaluatimi  Studies.” 

7.  On  page  21,  (section  entitled  “Methodological  Concent  Could  Adversely  Affect 
Evaluation  Studies”),  the  title  of  the  section  and  the  "assessment  of  the  evaluation" 
section  of  the  10  impact  studies  examined  in  the  appendix  of  the  report,  suggests  there  is 
no  room  for  limitations  in  studies.  There  are  ALWAYS  limitations  in  evaluation  and 
research  projects.  Indeed,  part  of  the  discussion  in  the  final  r^ort/summaiy  of  peer^ 
reviewed  studies  cautions  the  reader  to  attend  to  the  possible  limitations  of  interpreting 
the  study  findings  (i.e.,  sample  size,  generalizability).  There  is  a  significant  body  of 
research  that  cautions  the  researcher  in  utilizing  comparisons  across  sites  on  the  basis  of 
descriptive  data,  in  this,  the  case  number  of  child  abuse-founded  cases  (just  taking  one 
unit  of  measurement  fw  purposes  of  discussion).  Differences  between  communities  can 
make  any  type  of  comparison  questionable  at  best.  Prior  rate  of  child  abuse/neglect  in 
comparison  communities,  the  amount  of  resources  entering  the  community,  etc.,  cannot 
be  controlled.  A  number  of  writers  (Mark,  et  al.,  2000),  Henry,  Dickey,  &  Areso^  1991) 
have  documented  the  limitarions  of  attributing  program  effects  on  the  basis  of  unit  (e.g., 
site)  comparison.  It  is  simply  not  possible  to  control  for  everything  when  a  program  is 
administered  in  one  location  and  not  administered  in  another.  In  simplest  terms,  any 
observed  differaaces  between  a  demonstration  site  and  a  comparison  may  be  attnbutable 
not  only  to  the  absence  of  the  program,  but  to  the  differences  between  the  two  sites. 

There  is  no  way  to  accurately  attribute  impact  of  the  program  in  a  comparison  strategy 
that  spears  to  be  suggested  in  the  GAO  report 
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One  approach  to  program  evaluation  that  has  received  significant  respect  in  the  social 
science  field  and  would  likely  be  considered  a  “generally  accepted  standard,”  is 
comparisons  with  prior  data.  For  many  of  OJJDP  system-changing  initiatives,  knowing  if 
the  policy  being  implemented  is  moving  the  community  in  the  direction  we  hope  they 
would  go...such  as  increased  awareness  of  child  abuse/neglect  issues  in  the  community  or 
decrease  in  child  abuse/neglect  convictions,  is  a  primary  goal  of  the  evaluation.  Some 
methodologists  (i.e.,  Behn,  1997)  suggest  that  communities  should  target  perfonnance 
levels  that  are  customized  for  each  community.  This  targetii^  is  an  effective  tool  in 
motivating  communities  toward  performance  improvement.  Monitoring  perfonnance 
indicators  is  considered,  by  many  folks  in  the  field,  to  be  one  of  the  richest  sources  of 
comparison  analyses  for  evaluation  progress.”  Please  consider  the  following  references 
provided  by  the  OJJDP’s  RPDD. 

References  Supporting  RPDD  *s  Concerns  for  GAO’s  Consideration 

Behn,  R.  D  (1997).  Linking  Measurement  and  Motivation:  A  Challenge  for  Education. 
Advances  in  Educational  Administration,  5.  15-18. 

The  Annie  E.  Casey  Foundation  (1998)  Evaluating  Comprehensive  Community  Change, 
Baltimore,  MD:  Author 

Connell.  J,P.,  &  Kubisch.  A,C.  (1998),  Applying  a  Theory  of  Change  j^proach  to  the 
Evaluation  of  Comprehensive  Community  Initiatives:  Progress,  Prospects,  and  Problems. 

Henry,  G.  T„  Kickey,  K.C,  &Areson,  JC  (1991).  Stakeholder  Participation  in 
Educational  Performance  Monitoring.  Educational  Evaluation  and  Policy  Analysis, 
11(2),  177-188. 

K.  Fulbright-Anderson,  A.C.  Kubisch  &J.P.  Connell  (eds.).  New  Approaches  to 
Evaluating  Community  Initiatives:  Theory.  Measurement  and  Analysis.  Queenstown, 

Md:  The  Aspen  Institute. 

Knapp,  M.S.,  (1995).  How  Shall  We  Study  Comprehensive  Collaborative  Services  for 
Children  and  Families.  Educational  Researcher.  24(4),  5-1 6 

Mark,  M.  Henry,  G.,  &  Mnes,  G.  (2001).  Evaluation:  an  Integrated  Framework  for 
Understanding.  Guiding,  and  Improvins  Public  and  Nonprofit  Policies  <md  Programs. 
San  Francisco:  Jossey-bass. 

Mittlemark,  MB,,  Hunt.  M.K,  Heath,  G.W.  &  Schmid,  T.L  (1993).  Realistic  Outcomes: 
Lessons  from  Community-based  Research  and  Demonstration  Programs  for  the 
Prevention  of  Cardiovascular  Diseases.  Journal  of Public  Health  Policy,  1 4(4),  437-462. 
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Now  on  p.  25. 


Robinson,  G.H  d  Hill,  J.,  (1995).  Problems  in  the  Evaluation  of  Community-wide 
Initiatives.  New  Approaches  to  Evaluating  Community  Initiatives:  Concepts,  Methods, 
and  Context.  The  Aspen  Institute. 

Shaw,  KM,  Replogle,  E,  (1996).  Challenges  in  Evaluating  School-linked  Services: 
Toward  a  More  Comprehensive  Evaluation  Framework.  Evaluation  Review.  20(4) 

8.  On  pages  23-24  of  the  report,  GAO  states  that  the  Juvenile  Mentoring  Program  (JUMP) 

evaluation  ^‘compares  youth  entering  the  program  to  those  completing  it  However,  a 
vari^  of  other  factors,  including  the  feet  that  youth  in  the  program  are  likely  to  mature 
and,  thus,  improve  somewhat  spontaneously,  cannot  be  ruled  out  as  a  rival  cause  of 
change  feom  the  beginning  to  the  end  of  the  program.” 

In  response,  the  design  adopted  is  a  compromise  in  which  the  most  rigorous  design  was 
found  to  be  incompatible  with  the  program  being  evaluated.  The  designs  which  are 
appropriate  for  the  JUMP  program  were  adjusted  and  implemented  in  such  a  way  as  to 
improve  their  validity. 

An  experimental  design,  with  youth  randomly  assigned  to  experimental  and  comparison 
groups,  was  considered,  but  found  to  be  inappropriate  for  the  JUMP  evaluation  for 
several  reasons.  One  reason  is  the  small  number  of  participants  we  are  seeing  at  many 
sites,  a  function,  ironically,  of  the  success  of  the  JU^  projects  in  targeting  fee  youth 
most  in  need  of  mentoring,  those  living  in  poverty  whose  living  and  schooling 
arrangements  may  be  unstable  and  transitory.  The  needier  the  clients,  fee  fewer  can  be 
served  by  fee  same  size  staff,  and  many  projects  are  not  able  to  serve  even  30  clients, 
severely  restricting  design  options  and  statistical  analysis.  Another  senous  difSculty  with 
a  random  assignment  design  is  that  it  makes  it  more  difficult  to  carefully  match  mentors 
and  mentees.  Mentors  are  always  in  short  supply  and  project  staff  review  candidate  youth 
to  best  match  characteristics  for  a  successful  relationship.  Random  assignment  would 
halve  fee  number  of  options  for  matching,  and  feus  reduce  to  some  extent  program 
effectiveness. 

The  current  design  is  more  rigorous  than  a  simple  pro-  and  post-  test  because  it  will 
permit  using  different  cohorts  at  a  site  as  comparisons  for  each  other.  Youth  entering  a 
mentoring  relationship  during  the  second  year  of  a  project  will  have  their  pre-mentoring 
measurements  compared  to  fee  post-mentoring  measures  of  the  first-year  youth.  This 
design  can  provide  considerable  validity  i^en  augmented  by  additional  evaluation 
methods  including  qualitative  studies,  special  analyses  of  selected  projects,  special 
studies  of  long-term  impacts  and  sustainability,  and  fee  many  analyses  ^^ch  can  be 
conducted  of  data  from  175  sites. 

The  maturation  effect,  mentioned  in  the  report,  is  generally  observed  as  worsening 
behavior  as  fee  12  to  13  year-old  youth  ^proach  their  most  crime-prone  years  ai^  are 
exposed  to  increasing  numbers  of  risk  factors.  The  evaluation  will  control  for  this  effect 
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by  comparing  the  outcomes  of  programs  serving  youth  of  varying  ages  and  through  some 
of  the  special  analyses  mentioned  above. 

Now  on  p.  25. 

See  comment  2. 

9. 

On  page  24,  the  report  states  that  the  ‘‘JUMP  evaluation  has  experienced  problems 
obtaining  behavioral  measures  and  school  peifonnancc  data  with  which  to  gauge 
program-driven  change.”  As  mentioned  above,  data  collection  problems  are  made  more 
difficult  by  the  fact  that  the  youth  being  served  by  this  program  are,  in  fact,  those  most  in 
need  of  mentoring— those  whose  living  situations  are  most  unstable.  These  participants 
drop  out  of  mentoring  relationships  and  schools  when  living  arrangements  change  or 
caregivers  move,  and  it  is  often  impossible  to  obtain  post  program  data  when  a  child  is 
taken  out  of  school,  often  out  of  the  area,  with  no  advanced  notice  or  preparatioa  OJJDP 
and  the  evaluation  grantee  conduct  a  very  labor  intensive  program  of  regul^  contact  with 
project  stafftoobtainthe  best  data  possible.  Each  project  is  provided  with  a 
computerized  management  information  system  which  makes  reporting  a  very  simple 
process.  Because  some  sites  are  unable  to  provide  complete  data  on  all  participants,  an 
enhanced  analysis  will  be  conducted  of  sites  in  which  the  best  data  is  available. 

Now  on  pp.  25-26. 

10, 

On  page  24,  the  report  states  that  the  “Intensive  Aftercare  evaluation  has  experienced 
survey  response  rate  shortfalls,  in  some  cases  obtaining  response  rates  of  less  than  30 
percent,  which  may  affect  the  validity  of  the  findings  ”  The  poor  response  rates  for  some 
elements  at  different  sites  were  particularly  disappointing  because  this  evaluation  was  a 
strong  random  assignment  design.  The  strategies  for  obtaining  adequate  data,  including 
site  visits  firom  technical  assistance  and  evaluation  staff  and  regular  data  reports  to  the 
projects,  turned  out  to  be  insufficient.  The  program  staff  who  were  requir^  to  collect 
data  regarding  service  provision  and  participant  behavior  in  the  program  did  not  give  data 
collection  adequate  priority  in  comparison  to  their  other  duties.  This  is  particularly  true 
of  data  regarding  the  comparison  groups. 

Now  on  p.  26. 

11, 

On  page  25,  “Conclusions”  section,  the  report  suggests  that  more  evaluations  using 
experiment^  or  quasi-experimcntal  evaluation  designs  should  he  funded.  Please  note, 
many  communities  reject  participation  in  programs  that  are  evaluated  in  this  way  tecause 
they  feel  tiiat  it  requires  them  to  purposely  exclude  youth  from  receiving  services,  i.e„ 
control  or  comparison  groups.  These  communities  believe  that  this  type  of  evaluation  is 
not  necessary. 

Now  on  p.  26. 

12. 

On  page  25,  the  GAO  conclusion  highlights  the  lack  of  a  comparison  group.  The  Teen 

Court  evaluation  has  a  comparison  group,  with  data  collection  solely  supported  with  a 
minimal  amount  of  funding  from  a  few  Federal  agencies. 

Now  on  p.  31 . 

13. 

On  page  28,  Table  3:  OJJDP  Funds  Awarded  by  Program  Area,  Fiscal  Year  1996 
through  2000,  the  funding  amounts  for  the  Rural  Gang  Initiative  (RGI)  are  inaccurate.  It 
appears  that  all  funds  awarded  to  the  National  Youth  Gang  Center  (NYGC)  were  included 
as  part  of  this  Initiative.  Although  NYGC  is  responsible  for  providing  training  and 
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technical  assistance  for  the  RGI,  NYGC  also  carries  out  a  variety  of  tasks  that  are 
UDielated  to  RGI  under  the  same  grant  number.  NYGC  first  received  funding  to  provide 
training  and  technical  assistance  for  RGI  in  FY1998.  NYGC  fim<hng  amounts  by  task  are 
included  in  the  table  below. 


Funding  for  the  NYGC  Under  Grant  Number  1995-JD-MU-KOO  l  (Original  ^ugh 


FY1998 

Base  funding 

$750,000 

$750,000 

$775,000 

Rural  Gang  Initiative  Training  &TcchnlcaI  Assistance 

$215,000 

$375,000 

$450,000 

National  Youth  Gang  Symposium 

$50,000 

N/A 

N/A 

Youtb  Gangs  in  Indian  Country 

N/A 

N/A 

$50,000 

Total 

$1,015,000 

$1,125,000 

$1,275,000 

All  remaining  funding  under  RGI  was  awarded  in  FY  1999  and  FY  2000  and  was  given 
directly  to  program  sites  or  the  evaluation  grantee.  This  information  is  included  in  table 
below. 


Task 

Grantee 

Grant  Number 

Award  Date 

Amount 

FY1999 

Program 

City  of  Mt.  Vernon 

1999-JD-FX-0004 

5/18/99 

$125,000 

Program 

C(Hjn^  of  Glenn, 
D^partmoit  of  Probation 

1999-JD-FX-0002 

5/5/99 

$125,000 

Program 

ChyofElkCiiy 

1999-JD-FX-0003 

5/13/99 

$115,208 

Program 

Longviefw  Police  Dept. 

1999-JD-FX-OOOl 

4/27/99 

$122,546 

Evaluation 

National  Council  on 

Crime  and  Delinquency 

1999-JD-FX-KOOl 

4/27/99 

$250,000 

Subtotal 

FY2000 

Program 

CHy  of  ML  Vernon 

I99<LJ1>FX-0004  S-1 

9/30/00 

$275,000 

Program 

County  of  Glenn, 
I>q>artment  of  Probation 

1999-JD-FX-(K102  S-1 

9/30/00 

$275,000 

Program 

City  of  Elk  City 

1999-JD-FX-0003  S-1 

2/1/01 

$99,473 

Program 

Longview  Police  Dept. 

1999-JD-FX-OOOl  S-1 

2/15/01 

$100,000 

Evaluation 

National  Council  on 

Crime  and  Delinquency 

1999-JD-FX-KOOl  S-1 

9/30/00 

$275,000 

Subtotal 

$1,024,473 
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Now  on  p.  35. 


Now  on  p.  35. 


Now  on  p.  36,  table  6. 


Now  on  p.  48. 


Now  on  pp.  48-51 


Based  on  the  information  in  the  tables  above,  the  amounts  presented  in  Table  3  on  page  28  of  the 
report  should  be: 


Demonstration  and  replication 
programs 

FY  1996 

FY  1997 

FY  1998 

FY1999 

FY2000 

Rural  Gang  Initiative 

(zero) 

(zero) 

$215,000 

$1,112,754 

$1,474,473 

14.  On  page  32,  the  sentence  describing  promotional  avenues  OJJDP  employs  to  get  the  word 
out  about  its  products  should  be  augmented  to  reflect  the  fijU  range  used.  We  recommend 
revising  as  follows  (changes  shown  in  bold):  OJJDP  promotes  products  through  the 
NCJRS  Catalogs  OJJDP’s  Juvenile  Justice  joumal,  the  NCJRS  and  OJJDP  websites, 
email  lists,  OJP  press  announcements,  conference  displays,  criminal/juvenBe  justice 
newsletters  and  journals,  and  flier  mailings. 

Also  on  page  32,  three  sentences  down  from  above  text,  we  recommend  the  following 
revision  (shown  in  bold)  to  the  statement  on  how  to  order  copies:  Individuals  can  also 
order  copies  of  publications  online  or  by  calling  the  Clearinghouse’s  toll-free  number. 

15.  On  page  33,  Table  12,  second  title  under  the  Intensive  Aftercare  Evaluation  heading 
(“Reintegrating  Juvenile  Offenders...”),  the  text  in  the  “Automatic  mailing”  column 
should  be  changed  from  “None”to  reflect  “Information  not  available”  or  similar  text 

1 6.  On  page  46  .“Performance  Data  for  Training  and  Technical  Assistance  Grantees,”  the 
draft  GAO  report  says,  “Neither  TTAD  or  STAD  has  imposed  additional  or  more  specific 
reporting  requirements  on  their  training  and  technical  assistance  providers,  and,  for  the 
most  part,  neither  has  CPD.”  This  statement  is  incorrect  The  Training  and  Technical 
Assitance  Division  (TTAD)  imposed  considerable  increases  in  reporting  requirements 
from  all  of  the  grantees  that  divide  up  the  Juvenile  Accountability  Incentive  Block  Grant 
(JAIBG)  funds  annually.  Grantees  receiving  JAIBG  funds  must  send  in  a  monthly  report 
which  is  required.  These  grantees  comprise  approximately  1/5  of  all  TTAD  grantees  and 
account  for  approximately  $5  million  annually. 

1 7.  On  pages  46  -  49 , “Performance  Data  for  Training  and  Technical  Assistance  Grantees,”  a 
reader  of  the  report  may  assume  that  each  of  the  OJJDP  divisions  providing  technical 
assistance  are  delivering  the  same  types  of  services.  The  draft  GAO  report  fiirfoet  makes 
the  assumption  that  a  grantee  is  the  same  whether  the  grantee  is  a  state,  tribe,  non-profit 
organization,  for-profit,  or  a  local  jurisdiction.  The  use  of  formula  grant  dollars  is  treated 
the  same  as  discretionary  funds.  In  addition,  on  page  28,  the  chart  combines  traimng  and 
technical  assistance  funding  under  the  National  Institute  for  Juv^e  Justice  and 
Delinquency  Prevention.  However,  this  presentation  does  not  highlight  the  importance  of 
traimng  and  technical  assistance. 
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Nowon  p.  51. 

18. 

On  page  49,  the  draft  GAO  report  indicates  that  there  is  not  a  common  definition  of 
training  and  technical  assistance.  The  TTAD  published  the  “Training,  Technical 

Assistance,  and  Evaluation  Protocols”  booklet  two  years  ago,  which  clearly  defines  both 
training  and  technical  assistance.  However,  this  guide  is  not  mentioned  in  the  draft  GAO 
report 

Now  on  p.  51 . 

See  comment  3. 

19. 

On  page  49,  the  section  title  (“Performance  Data  for  Research  Grantees”),  it  appears  that 
publications  are  being  used  as  a  measure  of  performance  for  research  grants.  This  may  be 
too  simplistic  an  approach.  If  one  proceeds  with  this  idea,  then  there  should  be  a  table 
with  number  of  publications  per  grant  or  project.  Instead,  the  report  shows  a  table  of 
publications  listed  by  topics.  It  is  misleading  to  use  "Causes  and  Correlates"  as  a  topic 
since  so  many  of  the  reports  could  more  accurately  fell  into  one  of  the  other  categories 
(which  are  not  clearly  defined  and  often  overly).  It  is  not  clear  what  the  purpose  of  these 
tables  are:  to  show  how  productive  our  grants  are  or  to  show  how  productive  OJJDP 
grantees  on  the  whole  are  on  certain  topics. 

Now  on  p.  53,  tables  1 0 
and  11. 

The  number  of  publications  (fiom  1993-Sept  2000)  and  the  numbers  in  Table  7  on  p^e 

51  of  the  report  seem  very  low  (especially  juvenile  courts  and  juvenile  offendere).  It  is 
unclear  v^diether  these  include  fact  sheets  and  bulletins  firom  the  National  Juvenile  Court 

Data  Archive.  Please  note,  the  list  provided  to  GAO  during  the  review  listing  non-OJJDP 
publications  was  for  “active”  grants  only  and  did  not  constitute  a  complete  list. 

Now  on  p.  59. 

20. 

On  page  53,  “Rural  Gang  Initiative  Evaluation  Components”  indicates  that  two  of  the 
four  sites  are  not  a  part  of  the  evaluation  “because  they  did  not  fiilly  implement  the 
model,”  This  suggests  that  the  grantees  foiled  to  comply  with  grant  requirements  or  that 
their  performance  was  insufficient.  In  fact,  these  two  grantees  did  not  implement  the 
model  because  ftieir  gang  crime  problem  did  not  warrant  the  model’s  jwescribed 
intervention  approach,  i.e.,  the  model  was  not  appropriate  for  these  sites  and  they  were 
not  tasked  with  its  implementation. 

Now  on  p.  57-58. 

21. 

On  page  54,  the  description  of  the  Violence  Prevention  (Project  PATHE)  evaluation 
should  updated  to  include  Charleston  and  Baltimore  as  replication  sites.  When  GAO  was 
conducting  its  review,  the  sites  for  this  evaluation  had  not  been  identified.  Also,  the 
project  now  focuses  on  middle  schools  only,  and  will  not  focus  on  high  schools. 

Fundamentally,  the  design  of  the  project  (experimental  and  control  schools)  has  not 
changed,  and  the  plan  for  both  process  and  outcome  evaluations  continues  pending 
available  funding.  All  of  the  funds  for  this  effort  came  fcom  the  Centers  for  Disease 

Control  (CDC)  via  an  Interagency  Agreement  (lAA)  and  OJJDP  made  the  grant  award 
(co-managed  by  CDC  and  OJJDP). 

Now  on  p.  66. 

22. 

On  page  56,  the  report  states  that  the  JUMP  evaluation  is  expected  to  conclude  on 

September  30, 2001 .  The  latest  award  sets  project  and  budget  end-dates  of  September  30, 
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2002.  This  evaluation  is  funded  with  a  Mentoring  set-aside  for  evaluation.  Funding  will 
drop  to  zero,  or  pretty  close  to  it,  if  the  set-aside  is  discontinued.  We  do  not  move  the 
project  period  forward  beyond  the  budget  period  as  this  may  imply  an  assurance  of  future 
funding.  However,  ending  the  evaluation  before  September  30, 2004,  would  waste 
funding  and  effort  already  invested. 

Now  on  p.  64. 

See  comment  4. 

23. 

On  p^e  58,  the  GAO  questions  the  validity  of  the  EUDL  evaluation  design  around  the 
participant  and  comparison  communities.  The  report  argues  that  because  there  are  no 
universal  activities  that  all  EUDL  grantees  must  initiate  as  part  of  the  program,  there  is  no 
way  of  achieving  a  legitimate  representative  sample.  While  tiiere  are  no  specific  program 
activities  (i.e.  compliance  check,  cops  in  shops  etc,.)  that  are  mandated  to  be  a  part  of 
every  EUDL  discretionary  program,  the  impact  evaluation  "will  be  able  to  measure 
impacts  on  several  program  areas  across  each  site.  This  will  provide  important 
information  to  the  field.  In  other  words,  we  may  not  be  evaluating  EUDL,  be^se  there 
is  no  specific  EUDL  program,  however,  within  tiie  parameters  of  EUDL  funding,  there 
are  a  number  of  activities/programs,  such  as  compliance  check  and  cops  in  shops,  that  we 
will  be  able  to  compare  with  a  sample  of  communities  that  have  not  received  EUDL 
funding.  This  is  still  important  to  die  field,  and  should  not  be  ignored. 

Now  on  p.  70. 

24. 

On  page  61,  the  report  states  that  the  “second  phase  covers  the  two  project  sites  -  one 
enhancement  and  one  planning  -  in  which  the  program  is  being  continued  after  the  end  of 
the  first  funding  cycle.”  The  two  projects  which  are  the  subject  of  the  second  phase  are 
the  two  which  received  additional  OJJDP  funding  to  add  elements  to  their  programs  to 
bring  them  closer  to  the  OJJDP  Community  Assessment  Centers  (CAC)  concept  Of  Ae 
two  projects  which  did  not  receive  additional  funding  (and  are  therefore  not  included  in 

Ae  second  phase  of  Ae  evaluation),  one  has  ceased  to  function  as  a  separate  project,  Ac 
oAer,  a  planning  site,  continues  to  move  toward  operational  status. 

Now  on  p.  71. 

25. 

On  page  66,  overall,  Ae  comments  reflected  in  Ae  draft  GAO  report  pertainmg  to  Ae 

Drug  Free  Communities  Support  Program  national  evaluation  are  accurate.  AlAo^, 
please  note  that  Ae  report  does  not  reflect  that  we  require  supplementary  information 
fiom  Fiscal  Year  (FY)  1998  and  FY 1999  grantees  in  the  form  of  a  Part  II  to  Aeir 

Categorical  Assistance  Progress  Report  (CAPR).  This  additional  infoi^on  supports 

Ae  national  evaluation.  Grantees  fi:om  FY  1998  and  1999,  must  submit  CAPR,  Part  I  and 

II,  to  boA  OJJDP  and  Caliber  Associates.  Grantees  from  FYs  2000-2002  report  to 

OJJDP  only  and  do  not  have  Ae  CAPR,  Part  II  requirement. 

Now  on  p.  73. 

26. 

On  page  63  and  64,  Ae  description  of  the  national  evaluation  of  SafeFutures  is 
incomplete  and  (only  very  recently)  outdated  wiA  respect  to  Ae  meAodology  being  used. 

Also,  the  report  focuses  only  on  Ae  YouA  and  Caregiver  Surveys,  vAdch  were  eliminated 
and  did  not  mention  a  major  component  of  Ae  evaluation,  Ae  Client  InAcator  Data  Base. 
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Now  in  app.  VII,  p.  68. 


Regarding  the  Youth  and  Caregiver  Surveys,  after  extensive  developmental  work  on  the 
instrument  and  attempts  to  implement  the  surv^  in  three  sites,  by  late  Spring  2001,  it 
became  apparent  that  survey  response  was  poor  and  would  not  produce  worthwhile 
results.  A  decision  was  made  to  discontinue  this  component  of  the  research  and  redirect 
remaining  resources  to  more  productive  efforts,  including  the  expanded  qualitative 
analyses,  and  process  reporting.  Several  factors  contributed  to  poor  response,  including 
outdated  contact  information  for  the  sampled  youth  and  service  provider  ambivalence 
about  the  perceived  intrusiveness  of  the  data  collection. 

Nevertheless,  GAO  did  not  address  a  major  component  of  the  national  evaluation  that 
collects  program  performance  data  on  program  operations  and  client  outcomes  through 
the  Client  Indicator  Data  Base.  The  local  programs  are  required  to  collect  extensive 
information  fixim  selected  SafeFutures  program  components  on  individual  participante’ 
risk  and  protective  fector  profiles;  youth’s  service  utilization  and  agencies’  coordination 
of  services  for  youth  during  the  course  of  their  involvement  in  the  SafeFutures  program. 
In  addition,  outcome  measures  regarding  youth’s  educational  commitment  (school 
attendance,  achievement  and  behavior);  their  involvement  in  delinquency  and  crime,  and 
any  changes  in  their  risk  profiles.  Analysis  of  these  data  will  provide  a  picture  on 
program  performance  in  three  key  areas:  reaching  the  intended  high  risk  youth  clientele, 
coordinating  services  for  youth  with  multiple  problems,  and  subsequent  school 
performance  and  involvement  In  Ae  juvenile  justice  system. 

Please  note  Aat  Ae  Client  Indicator  Data  Base  was  made  available  to  Ae  GAO  Auditors, 
and  formed  Ae  bas^  for  some  sites  in  respondii^  to  Ae  questions  regatAng  Ae  number 
of  youA  served. 

27.  In  Ae  >^pendix  ni-“Non-OJJDP-Funded  Programs:  10  Impact  Evaluations,”  Ae  Teen 
Courts  evaluation  is  listed  and  described  on  pages  68-69  as  an  evaluation  of  a  non-funded 
OJJDP  program.  OJJDP  has  an  extensive  national  program  on  this  and  it  is  Ae  only 
Federal  or  private  national  initiative  on  youth  courts.  It  is  admimstered  through  OJJDP’s 
TTAD.  Also,  throughout  Ae  document  Acre  are  references  that  Acre  are  only  10  impact 
evaluations  on  OJJDP  programs.  This  should  be  changed  to  “1 1”. 

28.  On  page  68,  Ae  draft  report  says,  “...rather  than  adults,  youA  are  in  in  Teen 

Court”  It  Aould  be  noted  that  adults  are  in  charge  m  Teen  Court  and  it  is  recommended 
that  Ae  sentence  be  revised  to  say,  “...A^  differ  from  formal  courts  m  that  young  people 
are  able  to  assist  m  the  community  decision  making  process  for  dealing  wiA  juvenile 
offenders.” 
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The  following  are  GAO’s  comments  on  the  Department  of  Justice’s 
October  15, 2001,  letter. 


GAO  Comments 


As  we  indicated  in  our  report,  impact  evaluations,  such  as  the  types 
that  OJJDP  is  funding,  can  encounter  difficult  design  and 
implementation  challenges.  (See  section  titled,  Evaluations  of  OJJDP 
Programs  are  Difficult  to  Successfully  Design  and  Implement,^  Also, 
we  are  aware  that  virtually  all  impact  evaluations  have  limitations. 
However,  where  possible,  impact  evaluations  should  be  designed  to 
mitigate  as  many  rival  explanations  of  program  effects  as  feasible,  and 
potential  limitations  of  the  chosen  research  design  should  be 
acknowledged. 


2.  Our  statement  that  the  Juvenile  Mentoring  Program  evaluation  “has 
experienced  problems  obtaining  behavioral  measures  and  school 
performance  data”  was  not  intended  to  criticize  the  evaluators’  level  of 
effort,  but  rather  to  indicate  that  their  inability  to  obtain  data  firom 
school  and  law  enforcement  officials  in  many  of  the  study  sites  makes 
it  more  difficult  to  evaluate  how  well  the  program  is  achieving  its 
objectives  of  diminishing  delinquency,  gang  involvement,  and  school 
failure.  While  enhanced  analysis  of  sites  with  the  best  data  may  be 
warranted,  it  does  not  overcome  the  problem  of  having  a  large  number 
of  sites  with  little  or  no  reliable  data  from  school  and  law  enforcement 
officials.  This  problem  was  explicitly  recognized  by  OJJDP  in  its 
November  2000  report. 


3.  During  the  course  of  our  review,  OJJDP  officials  told  us  that  one 
measure  of  a  research  grantee’s  performance  is  the  number  of 
products  the  grantee  has  published.  These  officials  provided  us  a 
listing  of  all  products  published  by  active  research  grantees  through 
OJJDP  and  private  publishers  as  a  direct  result  of  OJJDP-funded 
research.  We  summarized  these  voluminous  data  by  topic  to  facilitate 
the  presentation. 

4.  Our  report  points  out  that  the  Enforcing  the  Underage  Drinking  Laws 
Program  evaluation  documents  OJJDP  provided  to  us  were  not  clear 
on  whether  the  sites  chosen  were  representative.  Our  report  does  not 
suggest  “there  is  no  way  of  achieving  a  legitimate  representative 
sample.”  However,  we  agree  with  OJJDP  that  the  evaluation  may  not 
be  evaluating  the  Enforcing  the  Underage  Drinking  Laws  Program 
because  there  may  be  no  program  components  common  to  all  project 
cities.  The  Assistant  Attorney  General  states  that  the  evaluation  will  be 
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able  to  measure  impacts  on  several  program  areas  across  each  site. 
However,  our  point  is  that  the  evaluator’s  plan  to  aggregate  data  across 
sites  may  be  inappropriate  because  wide  variation  allowed  by  the 
program  means  that  program  activities  are  not  common  across  all 
sites.  Therefore,  interpreting  and  generalizing  results  may  be 
problematic. 
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Contact  one: 
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E-mail:  fraudnet@gao.gov 

1-800-424-5454  (automated  answering  system) 
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